2023-04-09 09:53:33,757 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9114bfcc-42bd-9802-05ab-0c25bc62528e 2023-04-09 09:53:33,772 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.TestAcidGuaranteesWithEagerPolicy timeout: 13 mins 2023-04-09 09:53:33,786 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-04-09 09:53:33,787 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9114bfcc-42bd-9802-05ab-0c25bc62528e/cluster_581cb602-ccd9-7b40-d298-a4df8bdcd7d5, deleteOnExit=true 2023-04-09 09:53:33,787 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-04-09 09:53:33,787 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9114bfcc-42bd-9802-05ab-0c25bc62528e/test.cache.data in system properties and HBase conf 2023-04-09 09:53:33,788 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9114bfcc-42bd-9802-05ab-0c25bc62528e/hadoop.tmp.dir in system properties and HBase conf 2023-04-09 09:53:33,788 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9114bfcc-42bd-9802-05ab-0c25bc62528e/hadoop.log.dir in system properties and HBase conf 2023-04-09 09:53:33,789 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9114bfcc-42bd-9802-05ab-0c25bc62528e/mapreduce.cluster.local.dir in system properties and HBase conf 2023-04-09 09:53:33,789 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9114bfcc-42bd-9802-05ab-0c25bc62528e/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-04-09 09:53:33,789 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-04-09 09:53:33,910 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-04-09 09:53:34,320 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-04-09 09:53:34,325 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9114bfcc-42bd-9802-05ab-0c25bc62528e/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-04-09 09:53:34,326 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9114bfcc-42bd-9802-05ab-0c25bc62528e/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-04-09 09:53:34,326 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9114bfcc-42bd-9802-05ab-0c25bc62528e/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-04-09 09:53:34,326 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9114bfcc-42bd-9802-05ab-0c25bc62528e/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-09 09:53:34,327 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9114bfcc-42bd-9802-05ab-0c25bc62528e/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-04-09 09:53:34,327 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9114bfcc-42bd-9802-05ab-0c25bc62528e/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-04-09 09:53:34,328 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9114bfcc-42bd-9802-05ab-0c25bc62528e/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-09 09:53:34,328 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9114bfcc-42bd-9802-05ab-0c25bc62528e/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-09 09:53:34,329 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9114bfcc-42bd-9802-05ab-0c25bc62528e/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-04-09 09:53:34,329 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9114bfcc-42bd-9802-05ab-0c25bc62528e/nfs.dump.dir in system properties and HBase conf 2023-04-09 09:53:34,330 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9114bfcc-42bd-9802-05ab-0c25bc62528e/java.io.tmpdir in system properties and HBase conf 2023-04-09 09:53:34,330 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9114bfcc-42bd-9802-05ab-0c25bc62528e/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-09 09:53:34,330 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9114bfcc-42bd-9802-05ab-0c25bc62528e/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-04-09 09:53:34,331 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9114bfcc-42bd-9802-05ab-0c25bc62528e/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-04-09 09:53:34,914 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-09 09:53:34,919 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-09 09:53:35,215 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-04-09 09:53:35,408 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-04-09 09:53:35,427 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-09 09:53:35,467 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-04-09 09:53:35,504 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9114bfcc-42bd-9802-05ab-0c25bc62528e/java.io.tmpdir/Jetty_localhost_46863_hdfs____.5foahv/webapp 2023-04-09 09:53:35,664 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:46863 2023-04-09 09:53:35,703 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-09 09:53:35,703 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-09 09:53:36,179 WARN [Listener at localhost/46631] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-09 09:53:36,285 WARN [Listener at localhost/46631] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-04-09 09:53:36,302 WARN [Listener at localhost/46631] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-09 09:53:36,310 INFO [Listener at localhost/46631] log.Slf4jLog(67): jetty-6.1.26 2023-04-09 09:53:36,315 INFO [Listener at localhost/46631] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9114bfcc-42bd-9802-05ab-0c25bc62528e/java.io.tmpdir/Jetty_localhost_46305_datanode____d0itgd/webapp 2023-04-09 09:53:36,416 INFO [Listener at localhost/46631] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:46305 2023-04-09 09:53:36,788 WARN [Listener at localhost/44563] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-09 09:53:37,347 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x5241326ba5bf1b3e: Processing first storage report for DS-65f0c8c8-c54c-497b-ac93-085f3f579b7c from datanode a694fe9c-ce8f-43d0-bbb2-d1fd0f9fcdee 2023-04-09 09:53:37,349 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x5241326ba5bf1b3e: from storage DS-65f0c8c8-c54c-497b-ac93-085f3f579b7c node DatanodeRegistration(127.0.0.1:39281, datanodeUuid=a694fe9c-ce8f-43d0-bbb2-d1fd0f9fcdee, infoPort=34043, infoSecurePort=0, ipcPort=44563, storageInfo=lv=-57;cid=testClusterID;nsid=613764613;c=1681034014990), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-04-09 09:53:37,351 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x5241326ba5bf1b3e: Processing first storage report for DS-f01d8845-6d5c-47de-9047-749c9f20d4e4 from datanode a694fe9c-ce8f-43d0-bbb2-d1fd0f9fcdee 2023-04-09 09:53:37,351 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x5241326ba5bf1b3e: from storage DS-f01d8845-6d5c-47de-9047-749c9f20d4e4 node DatanodeRegistration(127.0.0.1:39281, datanodeUuid=a694fe9c-ce8f-43d0-bbb2-d1fd0f9fcdee, infoPort=34043, infoSecurePort=0, ipcPort=44563, storageInfo=lv=-57;cid=testClusterID;nsid=613764613;c=1681034014990), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-04-09 09:53:37,371 DEBUG [Listener at localhost/44563] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9114bfcc-42bd-9802-05ab-0c25bc62528e 2023-04-09 09:53:37,450 INFO [Listener at localhost/44563] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9114bfcc-42bd-9802-05ab-0c25bc62528e/cluster_581cb602-ccd9-7b40-d298-a4df8bdcd7d5/zookeeper_0, clientPort=57816, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9114bfcc-42bd-9802-05ab-0c25bc62528e/cluster_581cb602-ccd9-7b40-d298-a4df8bdcd7d5/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9114bfcc-42bd-9802-05ab-0c25bc62528e/cluster_581cb602-ccd9-7b40-d298-a4df8bdcd7d5/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-04-09 09:53:37,467 INFO [Listener at localhost/44563] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=57816 2023-04-09 09:53:37,477 INFO [Listener at localhost/44563] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-09 09:53:37,480 INFO [Listener at localhost/44563] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-09 09:53:38,155 INFO [Listener at localhost/44563] util.FSUtils(471): Created version file at hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc with version=8 2023-04-09 09:53:38,156 INFO [Listener at localhost/44563] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/hbase-staging 2023-04-09 09:53:38,483 INFO [Listener at localhost/44563] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-04-09 09:53:38,981 INFO [Listener at localhost/44563] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-04-09 09:53:39,013 INFO [Listener at localhost/44563] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-09 09:53:39,014 INFO [Listener at localhost/44563] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-09 09:53:39,014 INFO [Listener at localhost/44563] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-09 09:53:39,014 INFO [Listener at localhost/44563] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-09 09:53:39,014 INFO [Listener at localhost/44563] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-09 09:53:39,170 INFO [Listener at localhost/44563] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-04-09 09:53:39,242 DEBUG [Listener at localhost/44563] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-04-09 09:53:39,338 INFO [Listener at localhost/44563] ipc.NettyRpcServer(120): Bind to /172.31.14.131:41863 2023-04-09 09:53:39,349 INFO [Listener at localhost/44563] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-09 09:53:39,351 INFO [Listener at localhost/44563] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-09 09:53:39,373 INFO [Listener at localhost/44563] zookeeper.RecoverableZooKeeper(93): Process identifier=master:41863 connecting to ZooKeeper ensemble=127.0.0.1:57816 2023-04-09 09:53:39,412 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:418630x0, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-09 09:53:39,415 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:41863-0x101685764600000 connected 2023-04-09 09:53:39,445 DEBUG [Listener at localhost/44563] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-09 09:53:39,446 DEBUG [Listener at localhost/44563] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-09 09:53:39,449 DEBUG [Listener at localhost/44563] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-09 09:53:39,457 DEBUG [Listener at localhost/44563] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=41863 2023-04-09 09:53:39,458 DEBUG [Listener at localhost/44563] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=41863 2023-04-09 09:53:39,458 DEBUG [Listener at localhost/44563] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=41863 2023-04-09 09:53:39,459 DEBUG [Listener at localhost/44563] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=41863 2023-04-09 09:53:39,460 DEBUG [Listener at localhost/44563] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=41863 2023-04-09 09:53:39,467 INFO [Listener at localhost/44563] master.HMaster(444): hbase.rootdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc, hbase.cluster.distributed=false 2023-04-09 09:53:39,545 INFO [Listener at localhost/44563] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-04-09 09:53:39,545 INFO [Listener at localhost/44563] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-09 09:53:39,545 INFO [Listener at localhost/44563] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-09 09:53:39,545 INFO [Listener at localhost/44563] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-09 09:53:39,545 INFO [Listener at localhost/44563] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-09 09:53:39,546 INFO [Listener at localhost/44563] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-09 09:53:39,551 INFO [Listener at localhost/44563] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-04-09 09:53:39,554 INFO [Listener at localhost/44563] ipc.NettyRpcServer(120): Bind to /172.31.14.131:42467 2023-04-09 09:53:39,556 INFO [Listener at localhost/44563] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-04-09 09:53:39,563 DEBUG [Listener at localhost/44563] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-04-09 09:53:39,564 INFO [Listener at localhost/44563] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-09 09:53:39,567 INFO [Listener at localhost/44563] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-09 09:53:39,568 INFO [Listener at localhost/44563] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:42467 connecting to ZooKeeper ensemble=127.0.0.1:57816 2023-04-09 09:53:39,572 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:424670x0, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-09 09:53:39,572 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:42467-0x101685764600001 connected 2023-04-09 09:53:39,573 DEBUG [Listener at localhost/44563] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-09 09:53:39,574 DEBUG [Listener at localhost/44563] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-09 09:53:39,575 DEBUG [Listener at localhost/44563] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-09 09:53:39,576 DEBUG [Listener at localhost/44563] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=42467 2023-04-09 09:53:39,576 DEBUG [Listener at localhost/44563] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=42467 2023-04-09 09:53:39,576 DEBUG [Listener at localhost/44563] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=42467 2023-04-09 09:53:39,577 DEBUG [Listener at localhost/44563] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=42467 2023-04-09 09:53:39,577 DEBUG [Listener at localhost/44563] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=42467 2023-04-09 09:53:39,579 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,41863,1681034018311 2023-04-09 09:53:39,589 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-09 09:53:39,591 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,41863,1681034018311 2023-04-09 09:53:39,613 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-09 09:53:39,613 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-09 09:53:39,613 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-09 09:53:39,614 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-09 09:53:39,615 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,41863,1681034018311 from backup master directory 2023-04-09 09:53:39,616 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-09 09:53:39,619 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,41863,1681034018311 2023-04-09 09:53:39,619 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-09 09:53:39,620 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-09 09:53:39,620 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,41863,1681034018311 2023-04-09 09:53:39,623 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-04-09 09:53:39,625 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-04-09 09:53:39,713 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/hbase.id with ID: d2e29537-8404-4390-83ce-ccf4f45fb316 2023-04-09 09:53:39,757 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-09 09:53:39,772 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-09 09:53:39,813 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x4aaf52c6 to 127.0.0.1:57816 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-09 09:53:39,845 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@706dfaf2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-09 09:53:39,870 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-09 09:53:39,872 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-04-09 09:53:39,889 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-04-09 09:53:39,889 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-04-09 09:53:39,891 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-09 09:53:39,895 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-09 09:53:39,896 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-09 09:53:39,928 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/MasterData/data/master/store-tmp 2023-04-09 09:53:40,357 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-09 09:53:40,358 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-09 09:53:40,358 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-09 09:53:40,358 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-09 09:53:40,358 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-09 09:53:40,358 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-09 09:53:40,358 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-09 09:53:40,358 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-09 09:53:40,361 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/MasterData/WALs/jenkins-hbase4.apache.org,41863,1681034018311 2023-04-09 09:53:40,387 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C41863%2C1681034018311, suffix=, logDir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/MasterData/WALs/jenkins-hbase4.apache.org,41863,1681034018311, archiveDir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/MasterData/oldWALs, maxLogs=10 2023-04-09 09:53:40,453 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39281,DS-65f0c8c8-c54c-497b-ac93-085f3f579b7c,DISK] 2023-04-09 09:53:40,464 DEBUG [RS-EventLoopGroup-3-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-09 09:53:40,541 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/MasterData/WALs/jenkins-hbase4.apache.org,41863,1681034018311/jenkins-hbase4.apache.org%2C41863%2C1681034018311.1681034020397 2023-04-09 09:53:40,542 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:39281,DS-65f0c8c8-c54c-497b-ac93-085f3f579b7c,DISK]] 2023-04-09 09:53:40,542 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-04-09 09:53:40,543 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-09 09:53:40,546 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-04-09 09:53:40,547 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-04-09 09:53:40,612 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-04-09 09:53:40,620 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-04-09 09:53:40,643 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-04-09 09:53:40,655 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-09 09:53:40,661 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-09 09:53:40,662 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-09 09:53:40,680 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-04-09 09:53:40,684 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-09 09:53:40,686 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59502313, jitterRate=-0.11334644258022308}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-09 09:53:40,686 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-09 09:53:40,688 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-04-09 09:53:40,711 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-04-09 09:53:40,711 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-04-09 09:53:40,713 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-09 09:53:40,715 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-04-09 09:53:40,749 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 33 msec 2023-04-09 09:53:40,749 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-04-09 09:53:40,776 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-04-09 09:53:40,783 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-04-09 09:53:40,809 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-04-09 09:53:40,813 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-09 09:53:40,816 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-04-09 09:53:40,821 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-04-09 09:53:40,826 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-04-09 09:53:40,829 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-09 09:53:40,830 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-04-09 09:53:40,831 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-04-09 09:53:40,844 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-04-09 09:53:40,851 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-09 09:53:40,851 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-09 09:53:40,851 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-09 09:53:40,852 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,41863,1681034018311, sessionid=0x101685764600000, setting cluster-up flag (Was=false) 2023-04-09 09:53:40,867 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-09 09:53:40,874 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-04-09 09:53:40,875 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,41863,1681034018311 2023-04-09 09:53:40,881 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-09 09:53:40,887 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-04-09 09:53:40,891 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,41863,1681034018311 2023-04-09 09:53:40,894 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.hbase-snapshot/.tmp 2023-04-09 09:53:40,982 INFO [RS:0;jenkins-hbase4:42467] regionserver.HRegionServer(952): ClusterId : d2e29537-8404-4390-83ce-ccf4f45fb316 2023-04-09 09:53:40,989 DEBUG [RS:0;jenkins-hbase4:42467] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-04-09 09:53:40,997 DEBUG [RS:0;jenkins-hbase4:42467] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-04-09 09:53:40,997 DEBUG [RS:0;jenkins-hbase4:42467] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-04-09 09:53:41,002 DEBUG [RS:0;jenkins-hbase4:42467] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-04-09 09:53:41,004 DEBUG [RS:0;jenkins-hbase4:42467] zookeeper.ReadOnlyZKClient(139): Connect 0x033291d8 to 127.0.0.1:57816 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-09 09:53:41,016 DEBUG [RS:0;jenkins-hbase4:42467] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@df56fce, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-09 09:53:41,017 DEBUG [RS:0;jenkins-hbase4:42467] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@9cafaa8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-09 09:53:41,042 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-04-09 09:53:41,047 DEBUG [RS:0;jenkins-hbase4:42467] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:42467 2023-04-09 09:53:41,053 INFO [RS:0;jenkins-hbase4:42467] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-04-09 09:53:41,053 INFO [RS:0;jenkins-hbase4:42467] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-04-09 09:53:41,053 DEBUG [RS:0;jenkins-hbase4:42467] regionserver.HRegionServer(1023): About to register with Master. 2023-04-09 09:53:41,053 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-09 09:53:41,054 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-09 09:53:41,054 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-09 09:53:41,054 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-09 09:53:41,054 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-04-09 09:53:41,054 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-09 09:53:41,054 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-09 09:53:41,054 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-09 09:53:41,056 INFO [RS:0;jenkins-hbase4:42467] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,41863,1681034018311 with isa=jenkins-hbase4.apache.org/172.31.14.131:42467, startcode=1681034019544 2023-04-09 09:53:41,060 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1681034051060 2023-04-09 09:53:41,063 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-04-09 09:53:41,066 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-04-09 09:53:41,067 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-04-09 09:53:41,072 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-09 09:53:41,077 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-04-09 09:53:41,080 DEBUG [RS:0;jenkins-hbase4:42467] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-04-09 09:53:41,086 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-04-09 09:53:41,086 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-04-09 09:53:41,087 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-04-09 09:53:41,087 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-04-09 09:53:41,087 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-09 09:53:41,089 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-04-09 09:53:41,091 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-04-09 09:53:41,091 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-04-09 09:53:41,094 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-04-09 09:53:41,095 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-04-09 09:53:41,097 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1681034021096,5,FailOnTimeoutGroup] 2023-04-09 09:53:41,097 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1681034021097,5,FailOnTimeoutGroup] 2023-04-09 09:53:41,097 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-09 09:53:41,097 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-04-09 09:53:41,099 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-04-09 09:53:41,099 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-04-09 09:53:41,137 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48993, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-04-09 09:53:41,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:41,169 DEBUG [RS:0;jenkins-hbase4:42467] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc 2023-04-09 09:53:41,169 DEBUG [RS:0;jenkins-hbase4:42467] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:46631 2023-04-09 09:53:41,169 DEBUG [RS:0;jenkins-hbase4:42467] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-04-09 09:53:41,174 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-09 09:53:41,174 DEBUG [RS:0;jenkins-hbase4:42467] zookeeper.ZKUtil(162): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:41,175 WARN [RS:0;jenkins-hbase4:42467] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-09 09:53:41,175 INFO [RS:0;jenkins-hbase4:42467] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-09 09:53:41,176 DEBUG [RS:0;jenkins-hbase4:42467] regionserver.HRegionServer(1947): logDir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/WALs/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:41,177 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,42467,1681034019544] 2023-04-09 09:53:41,188 DEBUG [RS:0;jenkins-hbase4:42467] zookeeper.ZKUtil(162): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:41,199 DEBUG [RS:0;jenkins-hbase4:42467] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-04-09 09:53:41,208 INFO [RS:0;jenkins-hbase4:42467] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-04-09 09:53:41,228 INFO [RS:0;jenkins-hbase4:42467] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-04-09 09:53:41,231 INFO [RS:0;jenkins-hbase4:42467] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-04-09 09:53:41,231 INFO [RS:0;jenkins-hbase4:42467] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-04-09 09:53:41,233 INFO [RS:0;jenkins-hbase4:42467] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-04-09 09:53:41,240 INFO [RS:0;jenkins-hbase4:42467] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-04-09 09:53:41,241 DEBUG [RS:0;jenkins-hbase4:42467] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-09 09:53:41,241 DEBUG [RS:0;jenkins-hbase4:42467] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-09 09:53:41,241 DEBUG [RS:0;jenkins-hbase4:42467] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-09 09:53:41,241 DEBUG [RS:0;jenkins-hbase4:42467] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-09 09:53:41,242 DEBUG [RS:0;jenkins-hbase4:42467] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-09 09:53:41,242 DEBUG [RS:0;jenkins-hbase4:42467] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-09 09:53:41,242 DEBUG [RS:0;jenkins-hbase4:42467] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-09 09:53:41,242 DEBUG [RS:0;jenkins-hbase4:42467] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-09 09:53:41,242 DEBUG [RS:0;jenkins-hbase4:42467] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-09 09:53:41,243 DEBUG [RS:0;jenkins-hbase4:42467] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-09 09:53:41,244 INFO [RS:0;jenkins-hbase4:42467] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-04-09 09:53:41,244 INFO [RS:0;jenkins-hbase4:42467] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-04-09 09:53:41,244 INFO [RS:0;jenkins-hbase4:42467] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-04-09 09:53:41,260 INFO [RS:0;jenkins-hbase4:42467] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-04-09 09:53:41,263 INFO [RS:0;jenkins-hbase4:42467] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,42467,1681034019544-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-09 09:53:41,279 INFO [RS:0;jenkins-hbase4:42467] regionserver.Replication(203): jenkins-hbase4.apache.org,42467,1681034019544 started 2023-04-09 09:53:41,279 INFO [RS:0;jenkins-hbase4:42467] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,42467,1681034019544, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:42467, sessionid=0x101685764600001 2023-04-09 09:53:41,279 DEBUG [RS:0;jenkins-hbase4:42467] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-04-09 09:53:41,280 DEBUG [RS:0;jenkins-hbase4:42467] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:41,280 DEBUG [RS:0;jenkins-hbase4:42467] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,42467,1681034019544' 2023-04-09 09:53:41,280 DEBUG [RS:0;jenkins-hbase4:42467] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:41,281 DEBUG [RS:0;jenkins-hbase4:42467] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:41,281 DEBUG [RS:0;jenkins-hbase4:42467] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-04-09 09:53:41,281 DEBUG [RS:0;jenkins-hbase4:42467] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-04-09 09:53:41,281 DEBUG [RS:0;jenkins-hbase4:42467] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:41,282 DEBUG [RS:0;jenkins-hbase4:42467] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,42467,1681034019544' 2023-04-09 09:53:41,282 DEBUG [RS:0;jenkins-hbase4:42467] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-04-09 09:53:41,282 DEBUG [RS:0;jenkins-hbase4:42467] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-04-09 09:53:41,283 DEBUG [RS:0;jenkins-hbase4:42467] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-04-09 09:53:41,283 INFO [RS:0;jenkins-hbase4:42467] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-04-09 09:53:41,283 INFO [RS:0;jenkins-hbase4:42467] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-04-09 09:53:41,394 INFO [RS:0;jenkins-hbase4:42467] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C42467%2C1681034019544, suffix=, logDir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/WALs/jenkins-hbase4.apache.org,42467,1681034019544, archiveDir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/oldWALs, maxLogs=32 2023-04-09 09:53:41,413 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39281,DS-65f0c8c8-c54c-497b-ac93-085f3f579b7c,DISK] 2023-04-09 09:53:41,416 INFO [RS:0;jenkins-hbase4:42467] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/WALs/jenkins-hbase4.apache.org,42467,1681034019544/jenkins-hbase4.apache.org%2C42467%2C1681034019544.1681034021396 2023-04-09 09:53:41,417 DEBUG [RS:0;jenkins-hbase4:42467] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:39281,DS-65f0c8c8-c54c-497b-ac93-085f3f579b7c,DISK]] 2023-04-09 09:53:41,509 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-09 09:53:41,510 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-09 09:53:41,510 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc 2023-04-09 09:53:41,523 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-09 09:53:41,526 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-09 09:53:41,528 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/hbase/meta/1588230740/info 2023-04-09 09:53:41,529 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-09 09:53:41,530 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-09 09:53:41,531 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-09 09:53:41,533 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/hbase/meta/1588230740/rep_barrier 2023-04-09 09:53:41,534 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-09 09:53:41,534 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-09 09:53:41,535 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-09 09:53:41,537 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/hbase/meta/1588230740/table 2023-04-09 09:53:41,537 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-09 09:53:41,538 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-09 09:53:41,540 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/hbase/meta/1588230740 2023-04-09 09:53:41,541 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/hbase/meta/1588230740 2023-04-09 09:53:41,544 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-09 09:53:41,545 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-09 09:53:41,549 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-09 09:53:41,550 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60606007, jitterRate=-0.09690012037754059}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-09 09:53:41,550 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-09 09:53:41,550 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-09 09:53:41,550 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-09 09:53:41,550 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-09 09:53:41,550 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-09 09:53:41,550 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-09 09:53:41,551 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-09 09:53:41,551 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-09 09:53:41,557 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-04-09 09:53:41,557 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-04-09 09:53:41,564 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-04-09 09:53:41,576 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-04-09 09:53:41,578 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-04-09 09:53:41,730 DEBUG [jenkins-hbase4:41863] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-04-09 09:53:41,733 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,42467,1681034019544, state=OPENING 2023-04-09 09:53:41,740 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-04-09 09:53:41,743 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-09 09:53:41,744 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-09 09:53:41,746 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,42467,1681034019544}] 2023-04-09 09:53:41,931 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:41,934 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-04-09 09:53:41,938 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34312, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-04-09 09:53:41,948 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-04-09 09:53:41,949 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-09 09:53:41,953 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C42467%2C1681034019544.meta, suffix=.meta, logDir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/WALs/jenkins-hbase4.apache.org,42467,1681034019544, archiveDir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/oldWALs, maxLogs=32 2023-04-09 09:53:41,969 DEBUG [RS-EventLoopGroup-3-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39281,DS-65f0c8c8-c54c-497b-ac93-085f3f579b7c,DISK] 2023-04-09 09:53:41,974 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/WALs/jenkins-hbase4.apache.org,42467,1681034019544/jenkins-hbase4.apache.org%2C42467%2C1681034019544.meta.1681034021954.meta 2023-04-09 09:53:41,974 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:39281,DS-65f0c8c8-c54c-497b-ac93-085f3f579b7c,DISK]] 2023-04-09 09:53:41,974 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-04-09 09:53:41,976 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-04-09 09:53:41,992 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-04-09 09:53:41,997 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-04-09 09:53:42,003 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-04-09 09:53:42,003 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-09 09:53:42,003 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-04-09 09:53:42,003 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-04-09 09:53:42,006 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-09 09:53:42,008 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/hbase/meta/1588230740/info 2023-04-09 09:53:42,008 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/hbase/meta/1588230740/info 2023-04-09 09:53:42,008 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-09 09:53:42,009 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-09 09:53:42,009 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-09 09:53:42,011 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/hbase/meta/1588230740/rep_barrier 2023-04-09 09:53:42,011 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/hbase/meta/1588230740/rep_barrier 2023-04-09 09:53:42,011 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-09 09:53:42,012 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-09 09:53:42,013 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-09 09:53:42,014 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/hbase/meta/1588230740/table 2023-04-09 09:53:42,014 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/hbase/meta/1588230740/table 2023-04-09 09:53:42,014 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-09 09:53:42,015 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-09 09:53:42,017 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/hbase/meta/1588230740 2023-04-09 09:53:42,019 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/hbase/meta/1588230740 2023-04-09 09:53:42,023 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-09 09:53:42,025 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-09 09:53:42,027 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63535751, jitterRate=-0.05324353277683258}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-09 09:53:42,027 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-09 09:53:42,036 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1681034021924 2023-04-09 09:53:42,055 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-04-09 09:53:42,056 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-04-09 09:53:42,056 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,42467,1681034019544, state=OPEN 2023-04-09 09:53:42,059 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-04-09 09:53:42,059 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-09 09:53:42,063 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-04-09 09:53:42,064 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,42467,1681034019544 in 313 msec 2023-04-09 09:53:42,070 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-04-09 09:53:42,070 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 501 msec 2023-04-09 09:53:42,075 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 1.1170 sec 2023-04-09 09:53:42,075 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1681034022075, completionTime=-1 2023-04-09 09:53:42,075 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-04-09 09:53:42,076 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-04-09 09:53:42,137 DEBUG [hconnection-0x12cc5459-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-09 09:53:42,141 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34320, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-09 09:53:42,160 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-04-09 09:53:42,160 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1681034082160 2023-04-09 09:53:42,160 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1681034142160 2023-04-09 09:53:42,160 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 84 msec 2023-04-09 09:53:42,185 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,41863,1681034018311-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-09 09:53:42,185 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,41863,1681034018311-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-09 09:53:42,185 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,41863,1681034018311-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-09 09:53:42,187 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:41863, period=300000, unit=MILLISECONDS is enabled. 2023-04-09 09:53:42,187 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-04-09 09:53:42,193 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-04-09 09:53:42,202 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-04-09 09:53:42,204 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-09 09:53:42,213 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-04-09 09:53:42,215 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-04-09 09:53:42,219 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-09 09:53:42,241 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/hbase/namespace/e67a71a4b88e824863c78f9fe9bfd221 2023-04-09 09:53:42,244 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/hbase/namespace/e67a71a4b88e824863c78f9fe9bfd221 empty. 2023-04-09 09:53:42,245 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/hbase/namespace/e67a71a4b88e824863c78f9fe9bfd221 2023-04-09 09:53:42,245 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-04-09 09:53:42,670 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-04-09 09:53:42,672 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => e67a71a4b88e824863c78f9fe9bfd221, NAME => 'hbase:namespace,,1681034022203.e67a71a4b88e824863c78f9fe9bfd221.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp 2023-04-09 09:53:43,085 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1681034022203.e67a71a4b88e824863c78f9fe9bfd221.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-09 09:53:43,086 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing e67a71a4b88e824863c78f9fe9bfd221, disabling compactions & flushes 2023-04-09 09:53:43,086 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1681034022203.e67a71a4b88e824863c78f9fe9bfd221. 2023-04-09 09:53:43,086 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1681034022203.e67a71a4b88e824863c78f9fe9bfd221. 2023-04-09 09:53:43,086 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1681034022203.e67a71a4b88e824863c78f9fe9bfd221. after waiting 0 ms 2023-04-09 09:53:43,086 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1681034022203.e67a71a4b88e824863c78f9fe9bfd221. 2023-04-09 09:53:43,086 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1681034022203.e67a71a4b88e824863c78f9fe9bfd221. 2023-04-09 09:53:43,086 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for e67a71a4b88e824863c78f9fe9bfd221: 2023-04-09 09:53:43,091 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-04-09 09:53:43,108 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1681034022203.e67a71a4b88e824863c78f9fe9bfd221.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1681034023094"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681034023094"}]},"ts":"1681034023094"} 2023-04-09 09:53:43,136 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-09 09:53:43,138 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-09 09:53:43,143 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681034023138"}]},"ts":"1681034023138"} 2023-04-09 09:53:43,148 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-04-09 09:53:43,161 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=e67a71a4b88e824863c78f9fe9bfd221, ASSIGN}] 2023-04-09 09:53:43,165 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=e67a71a4b88e824863c78f9fe9bfd221, ASSIGN 2023-04-09 09:53:43,166 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=e67a71a4b88e824863c78f9fe9bfd221, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,42467,1681034019544; forceNewPlan=false, retain=false 2023-04-09 09:53:43,317 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=e67a71a4b88e824863c78f9fe9bfd221, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:43,318 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1681034022203.e67a71a4b88e824863c78f9fe9bfd221.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1681034023317"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681034023317"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681034023317"}]},"ts":"1681034023317"} 2023-04-09 09:53:43,324 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure e67a71a4b88e824863c78f9fe9bfd221, server=jenkins-hbase4.apache.org,42467,1681034019544}] 2023-04-09 09:53:43,484 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1681034022203.e67a71a4b88e824863c78f9fe9bfd221. 2023-04-09 09:53:43,484 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => e67a71a4b88e824863c78f9fe9bfd221, NAME => 'hbase:namespace,,1681034022203.e67a71a4b88e824863c78f9fe9bfd221.', STARTKEY => '', ENDKEY => ''} 2023-04-09 09:53:43,485 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace e67a71a4b88e824863c78f9fe9bfd221 2023-04-09 09:53:43,486 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1681034022203.e67a71a4b88e824863c78f9fe9bfd221.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-09 09:53:43,486 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for e67a71a4b88e824863c78f9fe9bfd221 2023-04-09 09:53:43,486 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for e67a71a4b88e824863c78f9fe9bfd221 2023-04-09 09:53:43,488 INFO [StoreOpener-e67a71a4b88e824863c78f9fe9bfd221-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region e67a71a4b88e824863c78f9fe9bfd221 2023-04-09 09:53:43,490 DEBUG [StoreOpener-e67a71a4b88e824863c78f9fe9bfd221-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/hbase/namespace/e67a71a4b88e824863c78f9fe9bfd221/info 2023-04-09 09:53:43,490 DEBUG [StoreOpener-e67a71a4b88e824863c78f9fe9bfd221-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/hbase/namespace/e67a71a4b88e824863c78f9fe9bfd221/info 2023-04-09 09:53:43,490 INFO [StoreOpener-e67a71a4b88e824863c78f9fe9bfd221-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e67a71a4b88e824863c78f9fe9bfd221 columnFamilyName info 2023-04-09 09:53:43,491 INFO [StoreOpener-e67a71a4b88e824863c78f9fe9bfd221-1] regionserver.HStore(310): Store=e67a71a4b88e824863c78f9fe9bfd221/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-09 09:53:43,493 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/hbase/namespace/e67a71a4b88e824863c78f9fe9bfd221 2023-04-09 09:53:43,494 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/hbase/namespace/e67a71a4b88e824863c78f9fe9bfd221 2023-04-09 09:53:43,498 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for e67a71a4b88e824863c78f9fe9bfd221 2023-04-09 09:53:43,501 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/hbase/namespace/e67a71a4b88e824863c78f9fe9bfd221/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-09 09:53:43,502 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened e67a71a4b88e824863c78f9fe9bfd221; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71074979, jitterRate=0.0590997189283371}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-09 09:53:43,502 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for e67a71a4b88e824863c78f9fe9bfd221: 2023-04-09 09:53:43,504 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1681034022203.e67a71a4b88e824863c78f9fe9bfd221., pid=6, masterSystemTime=1681034023477 2023-04-09 09:53:43,508 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1681034022203.e67a71a4b88e824863c78f9fe9bfd221. 2023-04-09 09:53:43,508 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1681034022203.e67a71a4b88e824863c78f9fe9bfd221. 2023-04-09 09:53:43,509 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=e67a71a4b88e824863c78f9fe9bfd221, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:43,510 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1681034022203.e67a71a4b88e824863c78f9fe9bfd221.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1681034023509"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681034023509"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681034023509"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681034023509"}]},"ts":"1681034023509"} 2023-04-09 09:53:43,517 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-04-09 09:53:43,517 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure e67a71a4b88e824863c78f9fe9bfd221, server=jenkins-hbase4.apache.org,42467,1681034019544 in 189 msec 2023-04-09 09:53:43,520 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-04-09 09:53:43,520 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=e67a71a4b88e824863c78f9fe9bfd221, ASSIGN in 356 msec 2023-04-09 09:53:43,521 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-09 09:53:43,522 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681034023522"}]},"ts":"1681034023522"} 2023-04-09 09:53:43,524 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-04-09 09:53:43,528 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-04-09 09:53:43,531 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 1.3230 sec 2023-04-09 09:53:43,619 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-04-09 09:53:43,622 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-04-09 09:53:43,622 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-09 09:53:43,662 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-04-09 09:53:43,680 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-09 09:53:43,686 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 35 msec 2023-04-09 09:53:43,696 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-04-09 09:53:43,706 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-09 09:53:43,711 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 15 msec 2023-04-09 09:53:43,721 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-04-09 09:53:43,726 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-04-09 09:53:43,726 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 4.106sec 2023-04-09 09:53:43,728 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-04-09 09:53:43,729 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-04-09 09:53:43,730 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-04-09 09:53:43,731 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,41863,1681034018311-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-04-09 09:53:43,731 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,41863,1681034018311-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-04-09 09:53:43,742 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-04-09 09:53:43,789 DEBUG [Listener at localhost/44563] zookeeper.ReadOnlyZKClient(139): Connect 0x6990bacd to 127.0.0.1:57816 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-09 09:53:43,795 DEBUG [Listener at localhost/44563] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@69cada93, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-09 09:53:43,809 DEBUG [hconnection-0x1701553e-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-09 09:53:43,822 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39840, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-09 09:53:43,834 INFO [Listener at localhost/44563] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,41863,1681034018311 2023-04-09 09:53:43,855 INFO [Listener at localhost/44563] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=207, OpenFileDescriptor=470, MaxFileDescriptor=60000, SystemLoadAverage=301, ProcessCount=169, AvailableMemoryMB=2520 2023-04-09 09:53:43,873 DEBUG [Listener at localhost/44563] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-09 09:53:43,877 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58476, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-09 09:53:43,888 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-09 09:53:43,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-09 09:53:43,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-09 09:53:43,898 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-09 09:53:43,901 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-09 09:53:43,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 9 2023-04-09 09:53:43,905 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29 2023-04-09 09:53:43,906 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29 empty. 2023-04-09 09:53:43,908 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29 2023-04-09 09:53:43,909 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-09 09:53:43,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-09 09:53:43,927 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-09 09:53:43,929 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 1732612c92cd83e5f8169e1b3b27bd29, NAME => 'TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp 2023-04-09 09:53:43,947 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-09 09:53:43,947 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 1732612c92cd83e5f8169e1b3b27bd29, disabling compactions & flushes 2023-04-09 09:53:43,947 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:43,947 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:43,947 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. after waiting 0 ms 2023-04-09 09:53:43,947 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:43,947 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:43,947 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 1732612c92cd83e5f8169e1b3b27bd29: 2023-04-09 09:53:43,952 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-09 09:53:43,953 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681034023953"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681034023953"}]},"ts":"1681034023953"} 2023-04-09 09:53:43,957 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-09 09:53:43,958 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-09 09:53:43,958 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681034023958"}]},"ts":"1681034023958"} 2023-04-09 09:53:43,961 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-09 09:53:43,970 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=1732612c92cd83e5f8169e1b3b27bd29, ASSIGN}] 2023-04-09 09:53:43,972 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=1732612c92cd83e5f8169e1b3b27bd29, ASSIGN 2023-04-09 09:53:43,973 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=1732612c92cd83e5f8169e1b3b27bd29, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,42467,1681034019544; forceNewPlan=false, retain=false 2023-04-09 09:53:44,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-09 09:53:44,125 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=1732612c92cd83e5f8169e1b3b27bd29, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:44,125 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681034024124"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681034024124"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681034024124"}]},"ts":"1681034024124"} 2023-04-09 09:53:44,128 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure 1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544}] 2023-04-09 09:53:44,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-09 09:53:44,288 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:44,288 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1732612c92cd83e5f8169e1b3b27bd29, NAME => 'TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.', STARTKEY => '', ENDKEY => ''} 2023-04-09 09:53:44,289 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 1732612c92cd83e5f8169e1b3b27bd29 2023-04-09 09:53:44,289 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-09 09:53:44,289 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1732612c92cd83e5f8169e1b3b27bd29 2023-04-09 09:53:44,289 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1732612c92cd83e5f8169e1b3b27bd29 2023-04-09 09:53:44,291 INFO [StoreOpener-1732612c92cd83e5f8169e1b3b27bd29-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 1732612c92cd83e5f8169e1b3b27bd29 2023-04-09 09:53:44,293 DEBUG [StoreOpener-1732612c92cd83e5f8169e1b3b27bd29-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A 2023-04-09 09:53:44,293 DEBUG [StoreOpener-1732612c92cd83e5f8169e1b3b27bd29-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A 2023-04-09 09:53:44,295 INFO [StoreOpener-1732612c92cd83e5f8169e1b3b27bd29-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-09 09:53:44,295 INFO [StoreOpener-1732612c92cd83e5f8169e1b3b27bd29-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1732612c92cd83e5f8169e1b3b27bd29 columnFamilyName A 2023-04-09 09:53:44,296 INFO [StoreOpener-1732612c92cd83e5f8169e1b3b27bd29-1] regionserver.HStore(310): Store=1732612c92cd83e5f8169e1b3b27bd29/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-09 09:53:44,296 INFO [StoreOpener-1732612c92cd83e5f8169e1b3b27bd29-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 1732612c92cd83e5f8169e1b3b27bd29 2023-04-09 09:53:44,298 DEBUG [StoreOpener-1732612c92cd83e5f8169e1b3b27bd29-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B 2023-04-09 09:53:44,298 DEBUG [StoreOpener-1732612c92cd83e5f8169e1b3b27bd29-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B 2023-04-09 09:53:44,298 INFO [StoreOpener-1732612c92cd83e5f8169e1b3b27bd29-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-09 09:53:44,299 INFO [StoreOpener-1732612c92cd83e5f8169e1b3b27bd29-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1732612c92cd83e5f8169e1b3b27bd29 columnFamilyName B 2023-04-09 09:53:44,300 INFO [StoreOpener-1732612c92cd83e5f8169e1b3b27bd29-1] regionserver.HStore(310): Store=1732612c92cd83e5f8169e1b3b27bd29/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-09 09:53:44,300 INFO [StoreOpener-1732612c92cd83e5f8169e1b3b27bd29-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 1732612c92cd83e5f8169e1b3b27bd29 2023-04-09 09:53:44,302 DEBUG [StoreOpener-1732612c92cd83e5f8169e1b3b27bd29-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C 2023-04-09 09:53:44,302 DEBUG [StoreOpener-1732612c92cd83e5f8169e1b3b27bd29-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C 2023-04-09 09:53:44,302 INFO [StoreOpener-1732612c92cd83e5f8169e1b3b27bd29-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-09 09:53:44,302 INFO [StoreOpener-1732612c92cd83e5f8169e1b3b27bd29-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1732612c92cd83e5f8169e1b3b27bd29 columnFamilyName C 2023-04-09 09:53:44,303 INFO [StoreOpener-1732612c92cd83e5f8169e1b3b27bd29-1] regionserver.HStore(310): Store=1732612c92cd83e5f8169e1b3b27bd29/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-09 09:53:44,304 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:44,306 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29 2023-04-09 09:53:44,306 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29 2023-04-09 09:53:44,310 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-09 09:53:44,312 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1732612c92cd83e5f8169e1b3b27bd29 2023-04-09 09:53:44,315 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-09 09:53:44,316 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1732612c92cd83e5f8169e1b3b27bd29; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72889429, jitterRate=0.08613713085651398}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-09 09:53:44,316 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1732612c92cd83e5f8169e1b3b27bd29: 2023-04-09 09:53:44,317 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29., pid=11, masterSystemTime=1681034024281 2023-04-09 09:53:44,320 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:44,320 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:44,321 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=1732612c92cd83e5f8169e1b3b27bd29, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:44,321 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681034024321"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681034024321"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681034024321"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681034024321"}]},"ts":"1681034024321"} 2023-04-09 09:53:44,326 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-04-09 09:53:44,327 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure 1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 in 196 msec 2023-04-09 09:53:44,330 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-04-09 09:53:44,330 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=1732612c92cd83e5f8169e1b3b27bd29, ASSIGN in 357 msec 2023-04-09 09:53:44,331 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-09 09:53:44,331 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681034024331"}]},"ts":"1681034024331"} 2023-04-09 09:53:44,334 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-09 09:53:44,338 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-09 09:53:44,341 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 446 msec 2023-04-09 09:53:44,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-09 09:53:44,526 INFO [Listener at localhost/44563] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 9 completed 2023-04-09 09:53:44,532 DEBUG [Listener at localhost/44563] zookeeper.ReadOnlyZKClient(139): Connect 0x6d61f291 to 127.0.0.1:57816 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-09 09:53:44,537 DEBUG [Listener at localhost/44563] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5f0e2360, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-09 09:53:44,541 DEBUG [Listener at localhost/44563] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-09 09:53:44,544 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39848, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-09 09:53:44,549 DEBUG [Listener at localhost/44563] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-09 09:53:44,552 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58492, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-09 09:53:44,562 DEBUG [Listener at localhost/44563] zookeeper.ReadOnlyZKClient(139): Connect 0x02df095a to 127.0.0.1:57816 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-09 09:53:44,569 DEBUG [Listener at localhost/44563] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7263b174, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-09 09:53:44,570 DEBUG [Listener at localhost/44563] zookeeper.ReadOnlyZKClient(139): Connect 0x61e72c30 to 127.0.0.1:57816 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-09 09:53:44,579 DEBUG [Listener at localhost/44563] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@748f869b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-09 09:53:44,580 DEBUG [Listener at localhost/44563] zookeeper.ReadOnlyZKClient(139): Connect 0x0e16d41b to 127.0.0.1:57816 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-09 09:53:44,584 DEBUG [Listener at localhost/44563] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@56223d42, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-09 09:53:44,584 DEBUG [Listener at localhost/44563] zookeeper.ReadOnlyZKClient(139): Connect 0x4fea27db to 127.0.0.1:57816 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-09 09:53:44,589 DEBUG [Listener at localhost/44563] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@30eb7fb2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-09 09:53:44,589 DEBUG [Listener at localhost/44563] zookeeper.ReadOnlyZKClient(139): Connect 0x7e728705 to 127.0.0.1:57816 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-09 09:53:44,594 DEBUG [Listener at localhost/44563] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2de4d1f6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-09 09:53:44,595 DEBUG [Listener at localhost/44563] zookeeper.ReadOnlyZKClient(139): Connect 0x0dbb9ac1 to 127.0.0.1:57816 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-09 09:53:44,599 DEBUG [Listener at localhost/44563] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@42927198, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-09 09:53:44,599 DEBUG [Listener at localhost/44563] zookeeper.ReadOnlyZKClient(139): Connect 0x49a45b6d to 127.0.0.1:57816 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-09 09:53:44,623 DEBUG [Listener at localhost/44563] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1818ad0a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-09 09:53:44,624 DEBUG [Listener at localhost/44563] zookeeper.ReadOnlyZKClient(139): Connect 0x04fb2b82 to 127.0.0.1:57816 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-09 09:53:44,628 DEBUG [Listener at localhost/44563] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@59d6b6ea, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-09 09:53:44,628 DEBUG [Listener at localhost/44563] zookeeper.ReadOnlyZKClient(139): Connect 0x3cd3abbd to 127.0.0.1:57816 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-09 09:53:44,633 DEBUG [Listener at localhost/44563] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6129209e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-09 09:53:44,640 DEBUG [hconnection-0x79e1e112-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-09 09:53:44,640 DEBUG [hconnection-0x635c9341-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-09 09:53:44,640 DEBUG [hconnection-0x1d94e186-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-09 09:53:44,644 DEBUG [hconnection-0x2b90131a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-09 09:53:44,645 DEBUG [hconnection-0x6018e13c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-09 09:53:44,645 DEBUG [hconnection-0x1865dd73-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-09 09:53:44,646 DEBUG [hconnection-0x2a12d69-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-09 09:53:44,648 DEBUG [hconnection-0x139a00b7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-09 09:53:44,650 DEBUG [hconnection-0x21233b14-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-09 09:53:44,651 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39864, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-09 09:53:44,651 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39866, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-09 09:53:44,653 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39880, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-09 09:53:44,655 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39904, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-09 09:53:44,656 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39938, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-09 09:53:44,660 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39892, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-09 09:53:44,664 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39920, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-09 09:53:44,664 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39926, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-09 09:53:44,668 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39954, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-09 09:53:44,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:53:44,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:53:44,696 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:53:44,696 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:44,697 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:53:44,697 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:53:44,697 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:44,697 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:44,699 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:44,699 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:44,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:44,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:44,700 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:44,700 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:53:44,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:44,700 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:44,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:53:44,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:44,701 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:53:44,703 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:53:44,710 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:53:44,710 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:44,712 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:53:44,713 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:53:44,715 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:53:44,713 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:44,715 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. started... 2023-04-09 09:53:44,724 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1732612c92cd83e5f8169e1b3b27bd29 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-09 09:53:44,727 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. as already flushing 2023-04-09 09:53:44,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 1732612c92cd83e5f8169e1b3b27bd29 2023-04-09 09:53:44,755 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=A 2023-04-09 09:53:44,755 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:53:44,756 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=B 2023-04-09 09:53:44,756 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:53:44,756 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=C 2023-04-09 09:53:44,756 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:53:44,966 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1681034024854, max timestamp=1681034024939Num uniques 859; 2023-04-09 09:53:44,966 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-09 09:53:44,997 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681034024877, max timestamp=1681034024954Num uniques 859; 2023-04-09 09:53:44,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-09 09:53:44,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681034024877, max timestamp=1681034024939Num uniques 859; 2023-04-09 09:53:44,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-09 09:53:45,029 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:45,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39920 deadline: 1681034085017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:45,035 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:45,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39864 deadline: 1681034085021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:45,037 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:45,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39904 deadline: 1681034085024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:45,038 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:45,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39926 deadline: 1681034085027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:45,038 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:45,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39954 deadline: 1681034085029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:45,163 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:45,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39864 deadline: 1681034085161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:45,164 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:45,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39954 deadline: 1681034085162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:45,165 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:45,165 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:45,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39926 deadline: 1681034085163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:45,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39920 deadline: 1681034085163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:45,170 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:45,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39904 deadline: 1681034085167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:45,319 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/A/2aa2e6bc555e493e8014f0b8d27cc471 2023-04-09 09:53:45,374 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:45,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39920 deadline: 1681034085370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:45,375 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:45,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39864 deadline: 1681034085371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:45,376 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:45,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39926 deadline: 1681034085373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:45,380 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:45,380 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:45,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39954 deadline: 1681034085370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:45,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39904 deadline: 1681034085379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:45,493 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/B/85f50da4b0584c62bdfca4d7eaeb0e51 2023-04-09 09:53:45,681 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:45,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39920 deadline: 1681034085677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:45,687 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:45,689 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:45,687 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:45,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39864 deadline: 1681034085679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:45,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39954 deadline: 1681034085683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:45,690 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:45,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39904 deadline: 1681034085685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:45,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39926 deadline: 1681034085681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:45,963 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/C/96ed90125b80487d81ff680cfd3a0fa9 2023-04-09 09:53:45,975 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/A/2aa2e6bc555e493e8014f0b8d27cc471 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/2aa2e6bc555e493e8014f0b8d27cc471 2023-04-09 09:53:45,985 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/2aa2e6bc555e493e8014f0b8d27cc471, entries=150, sequenceid=11, filesize=11.6 K 2023-04-09 09:53:45,990 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-04-09 09:53:45,994 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/B/85f50da4b0584c62bdfca4d7eaeb0e51 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/85f50da4b0584c62bdfca4d7eaeb0e51 2023-04-09 09:53:46,016 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/85f50da4b0584c62bdfca4d7eaeb0e51, entries=150, sequenceid=11, filesize=11.6 K 2023-04-09 09:53:46,021 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/C/96ed90125b80487d81ff680cfd3a0fa9 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/96ed90125b80487d81ff680cfd3a0fa9 2023-04-09 09:53:46,030 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/96ed90125b80487d81ff680cfd3a0fa9, entries=150, sequenceid=11, filesize=11.6 K 2023-04-09 09:53:46,037 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=135.63 KB/138880 for 1732612c92cd83e5f8169e1b3b27bd29 in 1313ms, sequenceid=11, compaction requested=false 2023-04-09 09:53:46,039 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-09 09:53:46,042 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1732612c92cd83e5f8169e1b3b27bd29: 2023-04-09 09:53:46,043 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:46,043 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:53:46,044 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:53:46,044 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,044 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:53:46,044 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:53:46,052 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:46,052 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:46,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:46,053 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:46,054 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:53:46,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:46,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:46,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:46,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:46,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:53:46,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a7e9652[Count = 0] remaining members to acquire global barrier 2023-04-09 09:53:46,057 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:53:46,057 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:46,062 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:46,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:46,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:46,062 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:53:46,062 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:53:46,062 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:53:46,062 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,062 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:53:46,070 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,070 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:53:46,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:46,071 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:46,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:46,071 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:53:46,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:46,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:46,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:46,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:46,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:46,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:53:46,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:53:46,080 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:53:46,080 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:53:46,080 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:53:46,080 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:53:46,080 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:53:46,086 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:46,086 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:46,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:46,086 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:46,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:46,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:46,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:46,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:46,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:46,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:46,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:46,091 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:46,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:46,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:46,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:46,098 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:46,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:46,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,116 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,116 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:46,116 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:46,116 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:46,116 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:46,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:46,116 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:46,116 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:46,116 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:53:46,117 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:46,117 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:46,117 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:46,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:53:46,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:46,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:46,120 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:53:46,120 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:53:46,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 1732612c92cd83e5f8169e1b3b27bd29 2023-04-09 09:53:46,210 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1732612c92cd83e5f8169e1b3b27bd29 3/3 column families, dataSize=142.33 KB heapSize=403.47 KB 2023-04-09 09:53:46,225 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:53:46,225 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=A 2023-04-09 09:53:46,226 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-09 09:53:46,226 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=B 2023-04-09 09:53:46,226 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-09 09:53:46,226 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=C 2023-04-09 09:53:46,226 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-09 09:53:46,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:53:46,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:53:46,267 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:46,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:53:46,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39926 deadline: 1681034086232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:53:46,270 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:53:46,270 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:46,271 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:46,273 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:53:46,273 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:53:46,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39954 deadline: 1681034086234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,274 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:46,274 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:46,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39904 deadline: 1681034086237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,274 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:46,274 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:46,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39920 deadline: 1681034086239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,276 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:46,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:46,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:46,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:46,278 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:46,278 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:53:46,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:46,279 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:53:46,279 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,279 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:53:46,284 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:46,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39864 deadline: 1681034086268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,305 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:53:46,306 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:53:46,306 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:53:46,306 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:46,307 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:53:46,316 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:53:46,316 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:53:46,316 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:46,316 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. started... 2023-04-09 09:53:46,329 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. as already flushing 2023-04-09 09:53:46,329 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:46,344 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:53:46,344 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:53:46,345 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,346 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:53:46,346 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:53:46,350 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,350 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:46,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:46,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:46,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:46,352 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:46,352 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:53:46,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:46,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:46,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:46,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:53:46,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5faddf99[Count = 0] remaining members to acquire global barrier 2023-04-09 09:53:46,355 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:53:46,355 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:46,356 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:46,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:46,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:46,357 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:53:46,357 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:53:46,357 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,358 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:53:46,357 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:53:46,360 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,360 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:53:46,360 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:46,360 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:53:46,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:46,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:46,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:46,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:46,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:46,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:46,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:46,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:53:46,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:53:46,370 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:53:46,370 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:53:46,370 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:53:46,370 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:53:46,370 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:53:46,376 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:46,376 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:46,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:46,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:46,376 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:46,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:46,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:46,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:46,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:46,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:46,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:46,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:46,380 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:46,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39904 deadline: 1681034086376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:46,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:46,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:46,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,377 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:46,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:46,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39926 deadline: 1681034086373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,380 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:46,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,388 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,395 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,395 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:46,395 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:46,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:46,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:46,395 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:46,398 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:46,398 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:46,397 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:46,395 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:46,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39920 deadline: 1681034086387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,395 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:53:46,399 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:53:46,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39864 deadline: 1681034086387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:46,398 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:46,399 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:53:46,400 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:46,400 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:46,400 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:53:46,400 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:46,400 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:46,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39954 deadline: 1681034086387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,500 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:53:46,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:53:46,505 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:53:46,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:53:46,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:53:46,520 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:53:46,520 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:46,521 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:53:46,521 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:53:46,524 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:46,525 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:46,528 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:46,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:46,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:46,528 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:46,529 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:46,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:53:46,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:46,530 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:53:46,531 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:53:46,531 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:53:46,531 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,531 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:53:46,531 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:53:46,531 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:46,532 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:53:46,536 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:53:46,536 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:53:46,536 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:46,536 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. started... 2023-04-09 09:53:46,536 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. as already flushing 2023-04-09 09:53:46,536 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:46,537 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:53:46,537 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:53:46,537 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,537 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:53:46,538 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:53:46,542 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:46,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:46,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:46,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:46,543 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:46,544 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:46,544 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:53:46,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:46,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:46,545 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:53:46,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1760c9be[Count = 0] remaining members to acquire global barrier 2023-04-09 09:53:46,546 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:53:46,546 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:46,547 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:46,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:46,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:46,547 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:53:46,547 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:53:46,547 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:53:46,549 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,549 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:53:46,550 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:53:46,550 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:46,551 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:53:46,552 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:46,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:46,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:46,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:46,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:46,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:46,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:46,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:53:46,559 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:53:46,559 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:53:46,559 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:53:46,559 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:53:46,559 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:53:46,559 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:53:46,563 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:46,563 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:46,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:46,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:46,565 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:46,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:46,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:46,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:46,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:46,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:46,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:46,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:46,567 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:46,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:46,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:46,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,569 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:46,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:46,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,583 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:46,583 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:46,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:46,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:46,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:46,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:46,584 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,584 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:46,584 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:46,584 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:46,585 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,585 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:46,585 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:46,585 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:46,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:53:46,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:53:46,589 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:53:46,589 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:53:46,591 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:46,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39926 deadline: 1681034086588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,591 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:46,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39904 deadline: 1681034086589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,603 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:46,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39920 deadline: 1681034086602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,604 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:46,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39864 deadline: 1681034086602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,605 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:46,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39954 deadline: 1681034086604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,689 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:53:46,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:53:46,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:53:46,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:53:46,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:53:46,698 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:53:46,698 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:46,700 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:53:46,700 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:53:46,700 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:46,701 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:46,702 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:46,702 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:46,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:46,703 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,703 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:53:46,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:46,704 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:46,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:53:46,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:46,705 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:53:46,705 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:53:46,705 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:53:46,705 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:53:46,706 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:46,708 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:53:46,708 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:53:46,708 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:53:46,708 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:46,709 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. started... 2023-04-09 09:53:46,709 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. as already flushing 2023-04-09 09:53:46,709 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:46,711 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:53:46,711 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:53:46,711 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,711 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:53:46,711 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:53:46,714 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:46,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:46,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:46,716 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:46,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:46,716 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:46,716 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:53:46,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:46,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:46,719 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:53:46,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ed46a78[Count = 0] remaining members to acquire global barrier 2023-04-09 09:53:46,720 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:53:46,720 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:46,722 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:46,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:46,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:46,722 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:53:46,722 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:53:46,722 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:53:46,728 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:53:46,729 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:46,729 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:53:46,728 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,728 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,729 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:53:46,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:46,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:46,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:46,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:46,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:46,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:46,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:46,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:53:46,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:53:46,737 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:53:46,737 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:53:46,737 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:53:46,737 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:53:46,737 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:53:46,739 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:46,739 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:46,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:46,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:46,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:46,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:46,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:46,739 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:46,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:46,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:46,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:46,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:46,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:46,742 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:46,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:46,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:46,745 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:46,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,755 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:46,755 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:46,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:46,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:46,755 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,755 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:46,755 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:46,755 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=55.91 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/A/a5cf92babdff45eb967f734cae0394be 2023-04-09 09:53:46,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:53:46,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:53:46,755 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,755 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:46,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:46,755 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:53:46,755 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:46,756 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:53:46,756 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:53:46,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:46,755 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:46,756 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:46,800 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.57 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/B/1d59c66533ce4ef3b859497853e9980a 2023-04-09 09:53:46,837 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.57 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/C/869cddd98f354b2fa8a89dceaf19a738 2023-04-09 09:53:46,848 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/A/a5cf92babdff45eb967f734cae0394be as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/a5cf92babdff45eb967f734cae0394be 2023-04-09 09:53:46,856 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:53:46,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:53:46,859 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/a5cf92babdff45eb967f734cae0394be, entries=150, sequenceid=49, filesize=11.6 K 2023-04-09 09:53:46,861 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/B/1d59c66533ce4ef3b859497853e9980a as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/1d59c66533ce4ef3b859497853e9980a 2023-04-09 09:53:46,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:53:46,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:53:46,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:53:46,884 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:53:46,884 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:46,888 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:53:46,889 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:53:46,889 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:46,889 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:46,890 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/1d59c66533ce4ef3b859497853e9980a, entries=150, sequenceid=49, filesize=11.6 K 2023-04-09 09:53:46,892 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/C/869cddd98f354b2fa8a89dceaf19a738 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/869cddd98f354b2fa8a89dceaf19a738 2023-04-09 09:53:46,896 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:46,896 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:46,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:46,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:46,897 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,897 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:53:46,898 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:46,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:53:46,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:46,898 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:53:46,900 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:46,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39904 deadline: 1681034086897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,901 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:46,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39926 deadline: 1681034086897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:46,907 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/869cddd98f354b2fa8a89dceaf19a738, entries=150, sequenceid=49, filesize=11.6 K 2023-04-09 09:53:46,908 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~149.04 KB/152620, heapSize ~424.05 KB/434224, currentSize=33.54 KB/34350 for 1732612c92cd83e5f8169e1b3b27bd29 in 698ms, sequenceid=49, compaction requested=false 2023-04-09 09:53:46,912 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:53:46,913 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1732612c92cd83e5f8169e1b3b27bd29: 2023-04-09 09:53:46,913 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:53:46,913 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:53:46,913 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:46,914 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:53:46,914 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:53:46,914 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:53:46,915 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:46,915 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. started... 2023-04-09 09:53:46,915 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1732612c92cd83e5f8169e1b3b27bd29 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-09 09:53:46,915 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=A 2023-04-09 09:53:46,915 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:53:46,915 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=B 2023-04-09 09:53:46,916 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:53:46,916 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=C 2023-04-09 09:53:46,916 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:53:46,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 1732612c92cd83e5f8169e1b3b27bd29 2023-04-09 09:53:46,925 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. as already flushing 2023-04-09 09:53:47,027 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681034026990, max timestamp=1681034027010Num uniques 859; 2023-04-09 09:53:47,027 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-09 09:53:47,056 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681034026990, max timestamp=1681034027010Num uniques 859; 2023-04-09 09:53:47,056 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-09 09:53:47,094 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:47,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39864 deadline: 1681034087088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:47,101 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:47,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39920 deadline: 1681034087093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:47,102 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:47,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39954 deadline: 1681034087095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:47,202 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:47,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39864 deadline: 1681034087197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:47,204 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:47,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39920 deadline: 1681034087204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:47,208 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:47,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39954 deadline: 1681034087208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:47,384 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=57 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/A/a771e3e4779e41b09857a4ad421e0a46 2023-04-09 09:53:47,409 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:47,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39926 deadline: 1681034087408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:47,410 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:47,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39864 deadline: 1681034087409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:47,411 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:47,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39920 deadline: 1681034087409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:47,415 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:47,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39954 deadline: 1681034087414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:47,420 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:47,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39904 deadline: 1681034087415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:47,448 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=57 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/B/a5a9d5c7512846cfa9a15639f7777380 2023-04-09 09:53:47,511 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=57 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/C/13066ace425b40aebe907451acebb346 2023-04-09 09:53:47,529 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/A/a771e3e4779e41b09857a4ad421e0a46 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/a771e3e4779e41b09857a4ad421e0a46 2023-04-09 09:53:47,541 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/a771e3e4779e41b09857a4ad421e0a46, entries=150, sequenceid=57, filesize=11.6 K 2023-04-09 09:53:47,545 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/B/a5a9d5c7512846cfa9a15639f7777380 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/a5a9d5c7512846cfa9a15639f7777380 2023-04-09 09:53:47,559 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/a5a9d5c7512846cfa9a15639f7777380, entries=150, sequenceid=57, filesize=11.6 K 2023-04-09 09:53:47,560 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/C/13066ace425b40aebe907451acebb346 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/13066ace425b40aebe907451acebb346 2023-04-09 09:53:47,581 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/13066ace425b40aebe907451acebb346, entries=150, sequenceid=57, filesize=11.6 K 2023-04-09 09:53:47,584 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=144.57 KB/148040 for 1732612c92cd83e5f8169e1b3b27bd29 in 669ms, sequenceid=57, compaction requested=true 2023-04-09 09:53:47,584 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1732612c92cd83e5f8169e1b3b27bd29: 2023-04-09 09:53:47,584 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:47,584 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:53:47,585 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:53:47,585 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:47,585 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:53:47,585 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:53:47,587 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:47,587 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:47,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:47,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:47,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:47,588 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:47,588 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:53:47,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:47,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:47,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:47,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:47,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:47,591 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:53:47,591 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3026b1f6[Count = 0] remaining members to acquire global barrier 2023-04-09 09:53:47,591 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:53:47,591 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:47,593 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:47,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:47,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:47,594 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:53:47,594 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:53:47,594 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:47,594 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:53:47,594 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:53:47,595 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:47,595 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:53:47,596 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:47,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:47,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:47,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:47,596 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:53:47,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:47,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:47,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:47,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:47,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:47,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:47,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:47,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:53:47,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:53:47,604 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:53:47,604 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:53:47,604 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:53:47,604 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:53:47,604 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:53:47,606 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:47,606 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:47,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:47,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:47,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:47,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:47,606 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:47,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:47,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:47,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:47,608 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:47,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:47,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:47,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:47,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:47,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:47,611 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:47,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:47,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:47,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:47,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:47,620 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:47,620 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:47,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:47,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:47,621 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:47,621 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:47,621 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:47,621 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:47,621 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:47,621 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:47,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:53:47,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:53:47,624 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-09 09:53:47,624 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:53:47,624 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:53:47,623 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:53:47,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:47,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:47,621 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:47,625 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:47,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 1732612c92cd83e5f8169e1b3b27bd29 2023-04-09 09:53:47,722 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1732612c92cd83e5f8169e1b3b27bd29 3/3 column families, dataSize=151.28 KB heapSize=441.25 KB 2023-04-09 09:53:47,722 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=A 2023-04-09 09:53:47,723 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:53:47,723 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=B 2023-04-09 09:53:47,723 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:53:47,723 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=C 2023-04-09 09:53:47,723 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:53:47,724 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:53:47,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:53:47,743 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:47,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39954 deadline: 1681034087728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:47,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:53:47,749 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:47,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39864 deadline: 1681034087743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:47,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:53:47,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:53:47,756 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:53:47,756 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:47,757 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-04-09 09:53:47,759 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-04-09 09:53:47,759 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:47,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39920 deadline: 1681034087756, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:47,772 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:53:47,772 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:53:47,783 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:47,784 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:47,789 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:47,789 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:47,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:47,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:47,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:47,790 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:47,790 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:53:47,790 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:47,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:53:47,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:47,791 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:53:47,791 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:53:47,792 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:53:47,792 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:53:47,792 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:47,792 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:53:47,800 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:53:47,800 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:53:47,800 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:47,800 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. started... 2023-04-09 09:53:47,800 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. as already flushing 2023-04-09 09:53:47,800 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:47,800 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:53:47,800 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:53:47,800 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:47,800 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:53:47,800 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:53:47,803 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:47,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:47,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:47,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:47,803 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:47,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:47,804 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:47,804 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:53:47,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:47,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:47,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:47,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:47,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:53:47,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c6e17c[Count = 0] remaining members to acquire global barrier 2023-04-09 09:53:47,806 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:53:47,806 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:47,808 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:47,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:47,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:47,808 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:53:47,808 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:53:47,808 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:47,808 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:53:47,808 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:53:47,810 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:53:47,810 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:47,810 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:47,810 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:53:47,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:47,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:47,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:47,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:47,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:47,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:47,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:47,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:47,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:47,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:47,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:53:47,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:53:47,816 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:53:47,817 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:53:47,817 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:53:47,817 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:53:47,817 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:53:47,819 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:47,819 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:47,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:47,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:47,819 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:47,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:47,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:47,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:47,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:47,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:47,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:47,820 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:47,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:47,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:47,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:47,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:47,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:47,826 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:47,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:47,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:47,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:47,831 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:47,831 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:47,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:47,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:47,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:47,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:47,833 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:47,833 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:47,833 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:47,833 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:47,833 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:47,833 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:47,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:53:47,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:53:47,833 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:47,833 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:47,833 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:53:47,837 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:53:47,837 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:53:47,856 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:47,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39954 deadline: 1681034087847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:47,857 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:47,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39864 deadline: 1681034087852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:47,868 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:47,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39920 deadline: 1681034087862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:47,937 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:53:47,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:53:47,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:53:47,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:53:47,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:53:47,953 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:53:47,954 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:47,957 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:53:47,957 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:53:47,958 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:47,958 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:47,960 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:47,960 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:47,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:47,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:47,962 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:47,962 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:53:47,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:47,963 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:47,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:53:47,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:47,964 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:53:47,965 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:53:47,965 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:53:47,965 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:53:47,965 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:47,969 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:53:47,969 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:53:47,969 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:53:47,969 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:47,969 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. started... 2023-04-09 09:53:47,969 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. as already flushing 2023-04-09 09:53:47,969 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:47,969 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:53:47,969 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:53:47,969 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:47,970 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:53:47,970 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:53:47,971 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:47,971 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:47,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:47,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:47,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:47,972 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:47,972 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:53:47,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:47,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:47,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:47,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:47,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:47,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:53:47,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@107af5c3[Count = 0] remaining members to acquire global barrier 2023-04-09 09:53:47,982 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:53:47,982 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:47,984 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:47,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:47,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:47,985 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:53:47,985 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:53:47,985 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:53:47,985 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:47,985 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:53:47,987 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:53:47,987 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:47,987 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:53:47,987 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:47,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:47,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:47,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:47,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:47,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:47,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:47,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:47,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:47,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:47,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:47,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:53:47,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:53:47,991 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:53:47,991 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:53:47,991 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:53:47,991 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:53:47,991 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:53:47,999 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:47,999 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:47,999 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:47,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:47,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:47,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:47,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:48,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:48,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:48,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:48,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:48,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:48,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:48,001 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:48,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:48,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:48,005 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:48,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,009 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,009 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:48,009 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:48,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:48,009 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:48,009 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:48,009 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,009 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:48,013 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:48,009 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:53:48,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:48,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:53:48,013 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:48,013 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:48,013 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:53:48,013 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:53:48,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:48,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:48,061 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:48,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39954 deadline: 1681034088060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,061 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:48,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39864 deadline: 1681034088060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,099 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:48,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39920 deadline: 1681034088071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,113 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:53:48,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:53:48,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:53:48,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:53:48,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:53:48,119 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:53:48,119 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:48,120 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:53:48,120 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:53:48,121 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:48,121 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:48,123 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:48,123 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:48,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:48,123 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,124 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:53:48,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:48,124 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:48,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:53:48,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:48,125 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:53:48,125 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:53:48,128 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:53:48,128 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:48,128 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:53:48,129 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:53:48,129 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:53:48,129 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:48,129 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. started... 2023-04-09 09:53:48,129 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. as already flushing 2023-04-09 09:53:48,129 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:48,129 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:53:48,129 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:53:48,129 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,129 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:53:48,129 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:53:48,131 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,131 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:48,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:48,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:48,131 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:48,131 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:53:48,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:48,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:48,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:48,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:48,134 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:53:48,134 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3dd0ca1e[Count = 0] remaining members to acquire global barrier 2023-04-09 09:53:48,134 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:53:48,134 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:48,136 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:48,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:48,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:48,137 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,137 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:53:48,137 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:53:48,137 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:53:48,137 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:53:48,139 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,139 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:53:48,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:48,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:48,139 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:48,139 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:53:48,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:48,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:48,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:48,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:48,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:48,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:53:48,144 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:53:48,144 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:53:48,144 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:53:48,144 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:53:48,144 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:53:48,144 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:53:48,147 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:48,147 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:48,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:48,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:48,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:48,147 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:48,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:48,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:48,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:48,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:48,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:48,149 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:48,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:48,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:48,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:48,158 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:48,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:48,164 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:48,164 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:48,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:48,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:48,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:48,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:48,166 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,166 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:48,166 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:48,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:53:48,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:53:48,166 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:48,167 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,167 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:48,167 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:48,167 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:48,167 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:53:48,168 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:53:48,168 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:53:48,191 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=39.86 KB at sequenceid=92 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/A/7a9fa2fab9474a13813123784afb3d91 2023-04-09 09:53:48,268 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:53:48,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:53:48,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:53:48,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:53:48,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:53:48,275 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:53:48,275 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:48,276 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:53:48,276 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:53:48,277 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:48,277 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:48,279 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:48,279 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:48,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:48,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:48,280 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,281 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:53:48,281 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:48,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:53:48,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:48,282 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:53:48,282 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:53:48,282 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:53:48,284 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:53:48,284 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:48,285 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:53:48,285 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:53:48,285 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:53:48,285 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:48,285 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. started... 2023-04-09 09:53:48,285 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. as already flushing 2023-04-09 09:53:48,285 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:48,286 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:53:48,286 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:53:48,286 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,286 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:53:48,286 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:53:48,290 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:48,290 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:48,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:48,290 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:48,290 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:53:48,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:48,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:48,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:48,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:48,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:53:48,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38fc7553[Count = 0] remaining members to acquire global barrier 2023-04-09 09:53:48,293 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:53:48,293 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:48,296 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:48,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:48,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:48,296 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:53:48,296 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:53:48,296 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,296 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:53:48,296 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:53:48,297 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:53:48,297 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,297 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:48,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:48,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:48,298 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:53:48,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:48,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:48,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:48,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:48,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:48,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:53:48,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:53:48,302 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:53:48,302 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:53:48,302 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:53:48,302 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:53:48,302 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:53:48,304 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:48,304 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:48,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:48,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:48,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:48,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:48,304 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:48,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:48,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:48,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:48,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:48,306 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:48,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:48,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:48,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:48,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,308 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:48,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:48,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,311 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,311 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:48,311 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:48,312 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,312 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:48,312 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:48,312 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:53:48,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:53:48,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:53:48,312 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:48,312 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:48,312 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:48,312 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:48,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:48,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:48,312 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:53:48,313 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:53:48,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:48,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:48,363 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:48,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39954 deadline: 1681034088363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,367 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:48,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39864 deadline: 1681034088365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,404 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:48,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39920 deadline: 1681034088403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,413 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:53:48,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:53:48,416 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:48,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39926 deadline: 1681034088414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:53:48,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:53:48,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:53:48,420 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:53:48,421 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:48,421 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:53:48,421 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:53:48,422 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:48,422 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:48,425 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:48,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:48,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:48,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:48,426 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:48,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:53:48,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:48,426 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:53:48,427 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,427 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:53:48,427 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:53:48,427 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:53:48,427 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:48,429 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:53:48,435 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:53:48,435 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:53:48,436 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:48,436 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. started... 2023-04-09 09:53:48,436 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. as already flushing 2023-04-09 09:53:48,436 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:48,436 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:53:48,436 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:53:48,436 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,436 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:53:48,436 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:53:48,439 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:48,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39904 deadline: 1681034088427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,439 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,439 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:48,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,439 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:48,439 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:53:48,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:48,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:48,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:48,440 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:48,441 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:53:48,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:48,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:48,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:53:48,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e07be31[Count = 0] remaining members to acquire global barrier 2023-04-09 09:53:48,443 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:53:48,443 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:48,444 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:48,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:48,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:48,445 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:53:48,445 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:53:48,445 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:53:48,445 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,445 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:53:48,448 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:53:48,448 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:48,448 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:48,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:48,448 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:53:48,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:48,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:48,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:48,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:48,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:48,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:53:48,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:53:48,455 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:53:48,455 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:53:48,455 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:53:48,455 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:53:48,455 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:53:48,458 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:48,458 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:48,458 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:48,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:48,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:48,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:48,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:48,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:48,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:48,459 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:48,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:48,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:48,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:48,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:48,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:48,461 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:48,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:48,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,464 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:48,464 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:48,464 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,464 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:48,464 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:48,464 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:48,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:48,464 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:53:48,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:53:48,464 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:48,465 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:48,465 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:48,465 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:53:48,465 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:53:48,465 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:53:48,465 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:48,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:48,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:48,479 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-09 09:53:48,480 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-04-09 09:53:48,565 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:53:48,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:53:48,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:53:48,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:53:48,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:53:48,570 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:53:48,570 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:48,571 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:53:48,571 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:53:48,572 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:48,572 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:48,573 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:48,573 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:48,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:48,574 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,574 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:53:48,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:48,574 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:48,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:53:48,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:48,575 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:53:48,575 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:53:48,575 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:53:48,575 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:53:48,576 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:48,577 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:53:48,578 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:53:48,578 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:53:48,578 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:48,578 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. started... 2023-04-09 09:53:48,578 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. as already flushing 2023-04-09 09:53:48,578 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:48,578 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:53:48,579 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:53:48,579 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,579 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:53:48,579 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:53:48,581 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,581 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:48,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:48,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:48,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:48,582 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:48,582 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:53:48,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:48,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:48,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:48,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:53:48,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@499486df[Count = 0] remaining members to acquire global barrier 2023-04-09 09:53:48,589 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:53:48,589 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:48,590 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:48,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:48,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:48,590 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:53:48,591 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:53:48,591 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:53:48,591 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,591 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:53:48,594 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:53:48,594 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:48,594 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:53:48,595 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:48,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:48,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:48,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:48,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:48,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:48,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:48,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:53:48,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:53:48,602 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:53:48,602 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:53:48,602 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:53:48,602 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:53:48,602 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:53:48,605 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:48,605 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:48,605 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:48,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:48,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:48,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:48,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:48,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:48,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:48,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:48,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:48,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:48,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:48,609 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:48,611 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:48,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,618 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,618 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:48,618 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:48,618 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:48,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:48,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:48,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:48,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:53:48,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:53:48,618 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,618 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:48,618 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:48,618 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:53:48,624 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:53:48,618 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:48,624 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:48,624 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:48,625 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:53:48,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:48,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:48,626 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=39.86 KB at sequenceid=92 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/B/634319912aa64e459118c58d3da2bb56 2023-04-09 09:53:48,676 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=71.56 KB at sequenceid=92 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/C/1e1ff3bb1e5c4962afdaca6efa4de26a 2023-04-09 09:53:48,686 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/A/7a9fa2fab9474a13813123784afb3d91 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/7a9fa2fab9474a13813123784afb3d91 2023-04-09 09:53:48,702 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/7a9fa2fab9474a13813123784afb3d91, entries=150, sequenceid=92, filesize=11.6 K 2023-04-09 09:53:48,706 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/B/634319912aa64e459118c58d3da2bb56 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/634319912aa64e459118c58d3da2bb56 2023-04-09 09:53:48,724 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:53:48,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:53:48,729 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:53:48,732 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/634319912aa64e459118c58d3da2bb56, entries=150, sequenceid=92, filesize=11.6 K 2023-04-09 09:53:48,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:53:48,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:53:48,733 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:53:48,733 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:48,735 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:53:48,735 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:53:48,739 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:48,739 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:48,739 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/C/1e1ff3bb1e5c4962afdaca6efa4de26a as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/1e1ff3bb1e5c4962afdaca6efa4de26a 2023-04-09 09:53:48,742 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,742 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:48,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:48,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:48,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:48,743 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,743 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:53:48,743 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:48,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:53:48,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:48,744 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:53:48,748 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:53:48,748 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:53:48,748 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:53:48,748 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:48,749 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:53:48,749 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:53:48,750 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:53:48,750 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:48,750 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. started... 2023-04-09 09:53:48,750 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. as already flushing 2023-04-09 09:53:48,750 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:48,750 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:53:48,750 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:53:48,750 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,750 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:53:48,750 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:53:48,754 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/1e1ff3bb1e5c4962afdaca6efa4de26a, entries=150, sequenceid=92, filesize=11.6 K 2023-04-09 09:53:48,754 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,754 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:48,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:48,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:48,755 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:48,755 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:53:48,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:48,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:48,755 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~151.28 KB/154910, heapSize ~441.20 KB/451792, currentSize=26.84 KB/27480 for 1732612c92cd83e5f8169e1b3b27bd29 in 1033ms, sequenceid=92, compaction requested=true 2023-04-09 09:53:48,756 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1732612c92cd83e5f8169e1b3b27bd29: 2023-04-09 09:53:48,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:48,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:48,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:53:48,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@722f35c9[Count = 0] remaining members to acquire global barrier 2023-04-09 09:53:48,757 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:53:48,758 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:48,759 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:48,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:48,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:48,759 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:53:48,760 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:53:48,760 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:53:48,760 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,760 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:53:48,762 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:53:48,762 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:48,762 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:53:48,764 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:48,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:48,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:48,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:48,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:48,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:48,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:48,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:53:48,767 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:53:48,767 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:53:48,767 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:53:48,768 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:53:48,768 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:53:48,768 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:53:48,769 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:53:48,772 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:53:48,772 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:53:48,772 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-09 09:53:48,772 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-09 09:53:48,774 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:48,774 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:48,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:48,774 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:48,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:48,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:48,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:48,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:48,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:48,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:48,778 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:48,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:48,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:48,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:48,780 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:48,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,781 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-09 09:53:48,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:48,781 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-09 09:53:48,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:48,783 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:48,788 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): 1732612c92cd83e5f8169e1b3b27bd29/B is initiating minor compaction (all files) 2023-04-09 09:53:48,788 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:48,788 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:48,788 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1732612c92cd83e5f8169e1b3b27bd29/B in TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:48,789 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/85f50da4b0584c62bdfca4d7eaeb0e51, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/1d59c66533ce4ef3b859497853e9980a, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/a5a9d5c7512846cfa9a15639f7777380, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/634319912aa64e459118c58d3da2bb56] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp, totalSize=46.4 K 2023-04-09 09:53:48,789 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:53:48,788 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1912): 1732612c92cd83e5f8169e1b3b27bd29/A is initiating minor compaction (all files) 2023-04-09 09:53:48,791 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 85f50da4b0584c62bdfca4d7eaeb0e51, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1681034024692 2023-04-09 09:53:48,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:53:48,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:53:48,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:48,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:48,788 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,792 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:48,792 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:48,792 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,792 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:48,792 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:48,792 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:48,792 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 1d59c66533ce4ef3b859497853e9980a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1681034025020 2023-04-09 09:53:48,791 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1732612c92cd83e5f8169e1b3b27bd29/A in TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:48,792 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:53:48,792 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:53:48,792 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/2aa2e6bc555e493e8014f0b8d27cc471, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/a5cf92babdff45eb967f734cae0394be, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/a771e3e4779e41b09857a4ad421e0a46, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/7a9fa2fab9474a13813123784afb3d91] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp, totalSize=46.4 K 2023-04-09 09:53:48,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:48,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:48,793 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting a5a9d5c7512846cfa9a15639f7777380, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=57, earliestPutTs=1681034026226 2023-04-09 09:53:48,794 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 634319912aa64e459118c58d3da2bb56, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=92, earliestPutTs=1681034027086 2023-04-09 09:53:48,796 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 2aa2e6bc555e493e8014f0b8d27cc471, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1681034024692 2023-04-09 09:53:48,803 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting a5cf92babdff45eb967f734cae0394be, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1681034025020 2023-04-09 09:53:48,804 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting a771e3e4779e41b09857a4ad421e0a46, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=57, earliestPutTs=1681034026226 2023-04-09 09:53:48,805 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 7a9fa2fab9474a13813123784afb3d91, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=92, earliestPutTs=1681034027086 2023-04-09 09:53:48,842 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): 1732612c92cd83e5f8169e1b3b27bd29#B#compaction#12 average throughput is 0.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:53:48,855 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1732612c92cd83e5f8169e1b3b27bd29#A#compaction#13 average throughput is 0.66 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:53:48,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 1732612c92cd83e5f8169e1b3b27bd29 2023-04-09 09:53:48,891 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1732612c92cd83e5f8169e1b3b27bd29 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-09 09:53:48,891 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=A 2023-04-09 09:53:48,892 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:53:48,892 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=B 2023-04-09 09:53:48,892 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:53:48,892 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=C 2023-04-09 09:53:48,892 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:53:48,892 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:53:48,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:53:48,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:53:48,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:53:48,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:53:48,913 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:53:48,914 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:48,916 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:53:48,917 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:53:48,924 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:48,924 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:48,926 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,926 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:48,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:48,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:48,933 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,933 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:53:48,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:48,938 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:48,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:53:48,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:48,939 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:53:48,940 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:53:48,940 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:53:48,940 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:53:48,940 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:48,943 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:53:48,948 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:53:48,948 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:53:48,948 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:48,948 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. started... 2023-04-09 09:53:48,948 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. as already flushing 2023-04-09 09:53:48,948 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:48,948 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:53:48,948 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:53:48,948 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,948 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:53:48,948 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:53:48,950 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/B/504f447e7bb54d5093883d200bb325f5 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/504f447e7bb54d5093883d200bb325f5 2023-04-09 09:53:48,951 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:48,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:48,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:48,953 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:48,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:48,959 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:48,959 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:53:48,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:48,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:48,961 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:53:48,961 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29a28032[Count = 0] remaining members to acquire global barrier 2023-04-09 09:53:48,961 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:53:48,961 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:48,963 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:48,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:48,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:48,963 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:53:48,963 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:53:48,963 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:53:48,963 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,964 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:53:48,965 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:48,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:48,966 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:53:48,966 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:48,966 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:53:48,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:48,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:48,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:48,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:48,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:48,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:53:48,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:53:48,971 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:53:48,971 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:53:48,971 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:53:48,971 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:53:48,971 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:53:48,972 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/A/f921476f471c463784c584e3a5ecbe41 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/f921476f471c463784c584e3a5ecbe41 2023-04-09 09:53:48,977 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:48,977 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:48,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:48,977 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:48,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:48,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:48,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:48,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:48,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:48,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:48,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:48,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:48,980 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:48,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:48,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:48,983 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:48,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:48,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,989 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,989 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:48,989 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:48,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:48,989 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:48,989 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:48,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:53:48,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:53:48,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:48,989 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:48,991 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:53:48,991 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:48,991 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:48,991 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:48,991 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:53:48,991 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:53:48,991 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:48,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:48,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:49,009 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1732612c92cd83e5f8169e1b3b27bd29/B of 1732612c92cd83e5f8169e1b3b27bd29 into 504f447e7bb54d5093883d200bb325f5(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:53:49,009 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1732612c92cd83e5f8169e1b3b27bd29/A of 1732612c92cd83e5f8169e1b3b27bd29 into f921476f471c463784c584e3a5ecbe41(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:53:49,009 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1732612c92cd83e5f8169e1b3b27bd29: 2023-04-09 09:53:49,009 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1732612c92cd83e5f8169e1b3b27bd29: 2023-04-09 09:53:49,009 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29., storeName=1732612c92cd83e5f8169e1b3b27bd29/B, priority=12, startTime=1681034028772; duration=0sec 2023-04-09 09:53:49,010 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:53:49,010 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-09 09:53:49,009 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29., storeName=1732612c92cd83e5f8169e1b3b27bd29/A, priority=12, startTime=1681034028760; duration=0sec 2023-04-09 09:53:49,011 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:53:49,014 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-09 09:53:49,014 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): 1732612c92cd83e5f8169e1b3b27bd29/C is initiating minor compaction (all files) 2023-04-09 09:53:49,014 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1732612c92cd83e5f8169e1b3b27bd29/C in TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:49,014 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/96ed90125b80487d81ff680cfd3a0fa9, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/869cddd98f354b2fa8a89dceaf19a738, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/13066ace425b40aebe907451acebb346, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/1e1ff3bb1e5c4962afdaca6efa4de26a] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp, totalSize=46.4 K 2023-04-09 09:53:49,015 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 96ed90125b80487d81ff680cfd3a0fa9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1681034024692 2023-04-09 09:53:49,016 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 869cddd98f354b2fa8a89dceaf19a738, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1681034025020 2023-04-09 09:53:49,020 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 13066ace425b40aebe907451acebb346, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=57, earliestPutTs=1681034026226 2023-04-09 09:53:49,023 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 1e1ff3bb1e5c4962afdaca6efa4de26a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=92, earliestPutTs=1681034027086 2023-04-09 09:53:49,046 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034029005, max timestamp=1681034029041Num uniques 859; 2023-04-09 09:53:49,046 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034029005, max timestamp=1681034029041Num uniques 859; 2023-04-09 09:53:49,052 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:53:49,052 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:53:49,053 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034029005, max timestamp=1681034029041Num uniques 859; 2023-04-09 09:53:49,053 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:53:49,062 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): 1732612c92cd83e5f8169e1b3b27bd29#C#compaction#15 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:53:49,092 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:53:49,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:53:49,095 WARN [DataStreamer for file /user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/C/0b4aef4af45c4703becfe47daf81bdc9] hdfs.DataStreamer(982): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) at java.lang.Thread.join(Thread.java:1331) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:630) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:807) 2023-04-09 09:53:49,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:53:49,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:53:49,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:53:49,101 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:53:49,101 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:49,104 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:53:49,105 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:53:49,105 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:49,105 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:49,107 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:49,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:49,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:49,107 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:49,108 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,108 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:53:49,108 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:49,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:53:49,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:49,109 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:53:49,114 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/C/0b4aef4af45c4703becfe47daf81bdc9 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/0b4aef4af45c4703becfe47daf81bdc9 2023-04-09 09:53:49,115 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:53:49,115 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:53:49,115 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:53:49,116 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:49,117 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:53:49,117 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:53:49,117 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:53:49,117 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:49,118 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. started... 2023-04-09 09:53:49,118 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. as already flushing 2023-04-09 09:53:49,118 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:49,118 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:53:49,118 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:53:49,118 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,118 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:53:49,118 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:53:49,120 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:49,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:49,120 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:49,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:49,121 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:49,121 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:53:49,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:49,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:49,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:49,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:53:49,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a675501[Count = 0] remaining members to acquire global barrier 2023-04-09 09:53:49,123 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:53:49,123 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:49,125 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:49,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:49,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:49,126 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:53:49,126 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:53:49,126 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:53:49,126 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,126 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:53:49,129 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:53:49,129 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,129 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:49,129 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:53:49,131 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:49,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39954 deadline: 1681034089120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,131 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:49,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39920 deadline: 1681034089122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:49,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:49,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:49,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:49,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:49,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:49,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:49,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:53:49,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:53:49,136 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:53:49,136 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:53:49,136 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:53:49,136 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:53:49,136 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:53:49,136 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:49,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39864 deadline: 1681034089131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,142 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1732612c92cd83e5f8169e1b3b27bd29/C of 1732612c92cd83e5f8169e1b3b27bd29 into 0b4aef4af45c4703becfe47daf81bdc9(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:53:49,143 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:49,143 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:49,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:49,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:49,143 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:49,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:49,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:49,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:49,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:49,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:49,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:49,144 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,144 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1732612c92cd83e5f8169e1b3b27bd29: 2023-04-09 09:53:49,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:49,145 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29., storeName=1732612c92cd83e5f8169e1b3b27bd29/C, priority=12, startTime=1681034028772; duration=0sec 2023-04-09 09:53:49,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:49,145 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:53:49,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:49,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:49,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,150 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:49,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:49,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,155 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:49,155 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:49,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:49,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:49,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:49,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:49,156 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,156 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:49,156 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:49,156 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:49,156 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:53:49,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:53:49,156 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,158 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:49,158 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:49,158 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:49,159 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:53:49,159 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:53:49,235 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:49,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39920 deadline: 1681034089234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,236 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:49,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39954 deadline: 1681034089234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,244 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:49,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39864 deadline: 1681034089243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,259 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:53:49,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:53:49,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:53:49,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:53:49,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:53:49,271 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:53:49,271 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:49,272 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:53:49,272 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:53:49,273 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:49,273 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:49,275 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:49,275 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:49,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:49,275 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,275 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:53:49,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:49,276 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:49,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:53:49,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:49,276 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:53:49,277 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:53:49,277 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:53:49,277 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:53:49,277 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:49,278 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:53:49,278 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:53:49,278 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:53:49,278 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:49,278 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. started... 2023-04-09 09:53:49,278 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. as already flushing 2023-04-09 09:53:49,278 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:49,278 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:53:49,278 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:53:49,278 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,279 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:53:49,279 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:53:49,280 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:49,280 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:49,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:49,281 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:49,281 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:53:49,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:49,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:49,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:49,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:49,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:53:49,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77289aec[Count = 0] remaining members to acquire global barrier 2023-04-09 09:53:49,285 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:53:49,285 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:49,289 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:49,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:49,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:49,290 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:53:49,290 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:53:49,290 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:53:49,293 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:53:49,293 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:49,293 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,293 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:53:49,294 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:53:49,294 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:53:49,295 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:53:49,295 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:53:49,295 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:53:49,295 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:53:49,295 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:53:49,296 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:49,296 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:49,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:49,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:49,297 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:49,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:49,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:49,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:49,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:49,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:49,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:49,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:49,299 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:49,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:49,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:49,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:49,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:49,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,303 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,307 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:49,307 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:49,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:49,307 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:49,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:53:49,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:53:49,307 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:49,307 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:49,307 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:49,307 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,307 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:49,313 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:49,313 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:49,313 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:53:49,313 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:53:49,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:49,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:49,360 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=102 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/A/198606809714499ca6fca91e0ce38388 2023-04-09 09:53:49,413 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:53:49,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:53:49,417 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:53:49,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:53:49,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:53:49,419 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:53:49,419 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:49,420 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:53:49,420 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:53:49,420 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:49,420 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:49,423 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,423 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:49,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:49,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:49,423 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:49,424 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:53:49,424 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:49,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:53:49,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:49,424 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:53:49,425 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:53:49,425 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:53:49,425 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:53:49,425 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:49,426 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:53:49,439 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:49,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39954 deadline: 1681034089438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,440 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:49,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39920 deadline: 1681034089438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,447 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:49,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39864 deadline: 1681034089446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,448 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:53:49,449 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:53:49,449 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:49,449 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. started... 2023-04-09 09:53:49,449 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. as already flushing 2023-04-09 09:53:49,449 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:49,449 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:53:49,449 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:53:49,449 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,450 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:53:49,450 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:53:49,451 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,451 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:49,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:49,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:49,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:49,452 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:49,452 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:53:49,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:49,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:49,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:49,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:53:49,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ca268bd[Count = 0] remaining members to acquire global barrier 2023-04-09 09:53:49,455 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:53:49,455 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:49,456 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:49,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:49,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:49,456 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:53:49,456 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:53:49,456 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:53:49,457 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,457 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:53:49,458 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:53:49,458 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:49,458 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:53:49,462 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:49,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:49,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:49,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:49,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:49,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:49,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:49,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:53:49,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:53:49,466 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:53:49,466 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:53:49,466 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:53:49,466 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:53:49,466 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:53:49,470 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:49,470 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:49,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:49,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:49,470 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:49,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:49,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:49,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:49,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:49,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:49,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:49,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:49,472 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:49,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:49,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:49,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:49,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:49,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,477 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,482 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:49,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:49,483 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,483 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:49,483 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:49,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:49,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:53:49,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:53:49,483 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:49,484 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:49,484 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,484 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:49,484 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:49,484 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:49,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:49,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:49,485 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:53:49,485 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:53:49,585 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:53:49,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:53:49,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:53:49,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:53:49,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:53:49,591 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:53:49,591 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:49,592 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:53:49,592 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:53:49,593 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:49,593 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:49,595 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:49,595 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:49,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:49,595 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,595 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:53:49,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:49,596 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:49,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:53:49,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:49,597 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:53:49,597 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:53:49,597 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:53:49,597 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:53:49,597 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:49,598 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:53:49,603 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:53:49,603 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:53:49,603 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:49,603 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. started... 2023-04-09 09:53:49,603 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. as already flushing 2023-04-09 09:53:49,603 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:49,604 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:53:49,604 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:53:49,604 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,604 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:53:49,604 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:53:49,605 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:49,606 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:49,606 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:53:49,606 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:49,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:49,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:49,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:49,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:49,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:49,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:53:49,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41bcd40b[Count = 0] remaining members to acquire global barrier 2023-04-09 09:53:49,609 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:53:49,609 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:49,610 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:49,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:49,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:49,610 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:53:49,611 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:53:49,611 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:53:49,614 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:53:49,614 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:49,614 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:53:49,616 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,616 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:53:49,617 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:53:49,617 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:53:49,617 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:53:49,617 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:53:49,617 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:53:49,617 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:53:49,620 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:49,620 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:49,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:49,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:49,621 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:49,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:49,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:49,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:49,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:49,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:49,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:49,621 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:49,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:49,625 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,628 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:49,628 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,628 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:49,628 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:49,628 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:49,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:53:49,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:53:49,628 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:49,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:49,632 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,632 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:49,632 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:49,632 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:53:49,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:49,632 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:53:49,632 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:49,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:49,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:49,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:49,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:49,634 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:53:49,733 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:53:49,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:53:49,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:53:49,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:53:49,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:53:49,742 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:53:49,742 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:49,743 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:53:49,743 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:53:49,744 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:49,744 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:49,746 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:49,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:49,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:49,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:49,747 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,747 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:49,747 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,747 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:53:49,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:53:49,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:49,748 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:53:49,748 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:49,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39954 deadline: 1681034089745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,749 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:49,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39920 deadline: 1681034089745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,752 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:49,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39864 deadline: 1681034089749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,753 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:53:49,753 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:53:49,753 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:53:49,753 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:49,754 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:53:49,754 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:53:49,754 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:53:49,754 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:49,754 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. started... 2023-04-09 09:53:49,755 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. as already flushing 2023-04-09 09:53:49,755 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:49,755 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:53:49,755 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:53:49,755 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,755 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:53:49,755 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:53:49,758 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,758 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:49,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:49,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:49,758 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:49,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:49,758 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:53:49,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:49,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:49,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:49,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:53:49,762 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61d35e60[Count = 0] remaining members to acquire global barrier 2023-04-09 09:53:49,762 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:53:49,762 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:49,763 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:49,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:49,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:49,763 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:53:49,763 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:53:49,763 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:53:49,764 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,764 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:53:49,765 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,765 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:53:49,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:49,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:49,765 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:49,765 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:53:49,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:49,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:49,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:49,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:49,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:49,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:53:49,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:53:49,769 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:53:49,769 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:53:49,769 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:53:49,769 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:53:49,769 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:53:49,774 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:49,774 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:49,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:49,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:49,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:49,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:49,774 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:49,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:49,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:49,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:49,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:49,775 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:49,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:49,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:49,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:49,776 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:49,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:49,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,779 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,779 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:49,779 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:49,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:49,779 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:49,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:53:49,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:53:49,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:49,779 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:49,779 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:49,780 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,780 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:49,780 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:49,780 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:53:49,780 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:53:49,780 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:49,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:49,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:49,808 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=102 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/B/6bba01064e294ec29be6740099bc6773 2023-04-09 09:53:49,832 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=102 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/C/d4962c47376c4769848588a5fbc9171c 2023-04-09 09:53:49,847 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/A/198606809714499ca6fca91e0ce38388 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/198606809714499ca6fca91e0ce38388 2023-04-09 09:53:49,858 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/198606809714499ca6fca91e0ce38388, entries=250, sequenceid=102, filesize=16.2 K 2023-04-09 09:53:49,861 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/B/6bba01064e294ec29be6740099bc6773 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/6bba01064e294ec29be6740099bc6773 2023-04-09 09:53:49,869 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/6bba01064e294ec29be6740099bc6773, entries=150, sequenceid=102, filesize=11.6 K 2023-04-09 09:53:49,878 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/C/d4962c47376c4769848588a5fbc9171c as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/d4962c47376c4769848588a5fbc9171c 2023-04-09 09:53:49,881 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:53:49,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:53:49,886 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:53:49,891 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/d4962c47376c4769848588a5fbc9171c, entries=150, sequenceid=102, filesize=11.6 K 2023-04-09 09:53:49,892 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 1732612c92cd83e5f8169e1b3b27bd29 in 1001ms, sequenceid=102, compaction requested=false 2023-04-09 09:53:49,893 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1732612c92cd83e5f8169e1b3b27bd29: 2023-04-09 09:53:49,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:53:49,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:53:49,896 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:53:49,896 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:49,897 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:53:49,897 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:53:49,898 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:49,898 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:49,899 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,899 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:49,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:49,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:49,900 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:49,900 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:53:49,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:49,900 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:49,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:53:49,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:49,901 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:53:49,901 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:53:49,901 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:53:49,901 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:53:49,901 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:49,904 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:53:49,906 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:53:49,906 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:53:49,912 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:49,912 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. started... 2023-04-09 09:53:49,912 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1732612c92cd83e5f8169e1b3b27bd29 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-09 09:53:49,913 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=A 2023-04-09 09:53:49,913 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:53:49,913 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=B 2023-04-09 09:53:49,913 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:53:49,913 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=C 2023-04-09 09:53:49,913 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:53:49,932 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=142 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/A/42572582c48b444ea0ae4f0fbc26d9c8 2023-04-09 09:53:49,959 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=142 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/B/cf656a65e8454cc69f66df09b0316251 2023-04-09 09:53:49,987 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=142 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/C/822298757f664773aab2cf90d9e91f83 2023-04-09 09:53:49,995 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/A/42572582c48b444ea0ae4f0fbc26d9c8 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/42572582c48b444ea0ae4f0fbc26d9c8 2023-04-09 09:53:50,003 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/42572582c48b444ea0ae4f0fbc26d9c8, entries=150, sequenceid=142, filesize=11.8 K 2023-04-09 09:53:50,006 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/B/cf656a65e8454cc69f66df09b0316251 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/cf656a65e8454cc69f66df09b0316251 2023-04-09 09:53:50,013 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/cf656a65e8454cc69f66df09b0316251, entries=150, sequenceid=142, filesize=11.8 K 2023-04-09 09:53:50,014 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/C/822298757f664773aab2cf90d9e91f83 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/822298757f664773aab2cf90d9e91f83 2023-04-09 09:53:50,031 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/822298757f664773aab2cf90d9e91f83, entries=150, sequenceid=142, filesize=11.8 K 2023-04-09 09:53:50,034 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=0 B/0 for 1732612c92cd83e5f8169e1b3b27bd29 in 121ms, sequenceid=142, compaction requested=true 2023-04-09 09:53:50,034 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1732612c92cd83e5f8169e1b3b27bd29: 2023-04-09 09:53:50,034 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:50,034 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:53:50,035 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:53:50,035 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:50,035 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:53:50,035 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:53:50,037 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:50,037 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:50,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:50,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:50,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:50,038 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:50,038 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:53:50,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:50,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:50,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:50,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:50,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:50,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:53:50,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@521f574c[Count = 0] remaining members to acquire global barrier 2023-04-09 09:53:50,042 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:53:50,042 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:50,044 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:50,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:50,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:50,045 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:53:50,045 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:53:50,045 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:53:50,046 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:50,046 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:53:50,047 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:53:50,047 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:50,047 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:53:50,049 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:50,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:50,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:50,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:50,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:50,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:50,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:50,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:50,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:50,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:50,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:50,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:53:50,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:53:50,053 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:53:50,053 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:53:50,053 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:53:50,053 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:53:50,053 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:53:50,055 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:50,055 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:50,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:50,055 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:50,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:50,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:50,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:50,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:50,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:50,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:50,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:50,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:50,058 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:50,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:50,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:50,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:50,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:50,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:50,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:50,061 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:50,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:50,065 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:50,065 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:50,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:50,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:50,066 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:50,066 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:50,066 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:50,066 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:50,066 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:50,066 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:50,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:50,066 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:50,066 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:50,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:53:50,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:53:50,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:50,066 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:53:50,066 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:53:50,167 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:53:50,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:53:50,172 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:53:50,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:53:50,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:53:50,181 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:53:50,181 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:50,185 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:53:50,185 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:53:50,185 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:50,185 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:50,187 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:50,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:50,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:50,187 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:50,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:50,188 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:50,188 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:50,188 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:53:50,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:53:50,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:50,189 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:53:50,192 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:53:50,192 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:53:50,192 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:53:50,192 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:50,200 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:53:50,201 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:53:50,201 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:53:50,201 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:50,201 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. started... 2023-04-09 09:53:50,201 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1732612c92cd83e5f8169e1b3b27bd29: 2023-04-09 09:53:50,201 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:50,201 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:53:50,201 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:53:50,201 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:50,202 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:53:50,202 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:53:50,203 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:50,203 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:50,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:50,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:50,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:50,204 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:50,204 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:53:50,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:50,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:50,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:50,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:50,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:50,207 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:53:50,207 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d973a32[Count = 0] remaining members to acquire global barrier 2023-04-09 09:53:50,207 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:53:50,208 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:50,209 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:50,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:50,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:50,209 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:53:50,209 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:53:50,209 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:53:50,209 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:50,209 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:53:50,212 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:53:50,212 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:50,212 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:53:50,212 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:50,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:50,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:50,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:50,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:50,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:50,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:50,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:50,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:50,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:50,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:50,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:53:50,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:53:50,217 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:53:50,218 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:53:50,218 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:53:50,218 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:53:50,218 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:53:50,220 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:50,220 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:50,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:50,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:50,220 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:50,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:50,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:50,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:50,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:50,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:50,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:50,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:50,221 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:50,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:50,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:50,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:50,224 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:50,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:50,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:50,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:50,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:50,227 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:50,227 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:50,227 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:50,227 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:50,228 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:50,228 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:50,228 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:53:50,228 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:50,232 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:50,232 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:50,232 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:53:50,234 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:50,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:53:50,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:50,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:50,235 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:53:50,235 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:53:50,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:50,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:50,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 1732612c92cd83e5f8169e1b3b27bd29 2023-04-09 09:53:50,265 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1732612c92cd83e5f8169e1b3b27bd29 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-09 09:53:50,265 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=A 2023-04-09 09:53:50,265 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:53:50,265 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=B 2023-04-09 09:53:50,266 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:53:50,266 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=C 2023-04-09 09:53:50,266 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:53:50,315 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=153 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/A/f1e1fd314ff84c439767c47794cc3b5c 2023-04-09 09:53:50,335 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:53:50,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:53:50,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:53:50,345 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034030317, max timestamp=1681034030332Num uniques 859; 2023-04-09 09:53:50,345 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034030317, max timestamp=1681034030338Num uniques 859; 2023-04-09 09:53:50,345 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:53:50,345 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:53:50,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:53:50,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:53:50,346 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:53:50,346 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:50,346 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:53:50,346 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:53:50,346 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034030317, max timestamp=1681034030338Num uniques 859; 2023-04-09 09:53:50,346 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:53:50,346 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:50,346 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:50,354 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:50,354 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:50,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:50,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:50,354 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:50,354 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:53:50,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:50,356 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:50,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:53:50,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:50,356 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:53:50,393 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:53:50,393 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:53:50,393 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:53:50,393 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:50,393 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:53:50,394 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:53:50,394 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:53:50,394 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:50,394 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. started... 2023-04-09 09:53:50,406 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. as already flushing 2023-04-09 09:53:50,407 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:50,407 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:53:50,407 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:53:50,407 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:50,407 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:53:50,407 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:53:50,415 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=153 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/B/3d9d91e6b6a04224b63c91c3bcd0e2c1 2023-04-09 09:53:50,416 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:50,416 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:50,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:50,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:50,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:50,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:50,417 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:50,417 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:53:50,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:50,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:50,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:50,420 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:50,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39920 deadline: 1681034090415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:50,421 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:50,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:50,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39864 deadline: 1681034090417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:50,422 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:50,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39954 deadline: 1681034090417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:50,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:53:50,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1aacad07[Count = 0] remaining members to acquire global barrier 2023-04-09 09:53:50,422 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:53:50,423 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:50,424 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:50,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:50,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:50,424 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:53:50,424 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:53:50,424 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:53:50,425 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:50,425 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:53:50,428 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:50,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:50,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:50,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:50,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:50,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:50,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:50,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:50,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:50,431 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:53:50,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:50,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:50,431 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:50,431 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:53:50,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:53:50,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:53:50,432 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:53:50,432 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:53:50,432 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:53:50,432 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:53:50,432 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:53:50,436 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:50,436 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:50,436 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:50,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:50,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:50,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:50,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:50,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:50,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:50,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:50,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:50,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:50,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:50,438 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:50,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:50,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:50,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:50,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:50,440 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:50,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:50,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:50,442 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:50,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39926 deadline: 1681034090439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:50,456 DEBUG [Thread-141] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4211 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29., hostname=jenkins-hbase4.apache.org,42467,1681034019544, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-09 09:53:50,456 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:50,456 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:50,457 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:50,465 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:50,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:50,456 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:50,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:50,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39904 deadline: 1681034090446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:50,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:50,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:50,467 DEBUG [Thread-143] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4230 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29., hostname=jenkins-hbase4.apache.org,42467,1681034019544, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-09 09:53:50,468 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:50,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:53:50,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:53:50,468 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:50,469 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:50,469 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:50,469 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:50,469 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:50,469 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:53:50,469 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:53:50,501 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=153 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/C/da1b99f29e1f4d5086a1fcb7f42ce6f0 2023-04-09 09:53:50,512 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/A/f1e1fd314ff84c439767c47794cc3b5c as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/f1e1fd314ff84c439767c47794cc3b5c 2023-04-09 09:53:50,523 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/f1e1fd314ff84c439767c47794cc3b5c, entries=250, sequenceid=153, filesize=16.4 K 2023-04-09 09:53:50,526 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/B/3d9d91e6b6a04224b63c91c3bcd0e2c1 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/3d9d91e6b6a04224b63c91c3bcd0e2c1 2023-04-09 09:53:50,529 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:50,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39920 deadline: 1681034090524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:50,529 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:50,530 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:50,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39864 deadline: 1681034090524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:50,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39954 deadline: 1681034090524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:50,537 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/3d9d91e6b6a04224b63c91c3bcd0e2c1, entries=100, sequenceid=153, filesize=9.4 K 2023-04-09 09:53:50,540 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/C/da1b99f29e1f4d5086a1fcb7f42ce6f0 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/da1b99f29e1f4d5086a1fcb7f42ce6f0 2023-04-09 09:53:50,553 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/da1b99f29e1f4d5086a1fcb7f42ce6f0, entries=100, sequenceid=153, filesize=9.4 K 2023-04-09 09:53:50,555 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 1732612c92cd83e5f8169e1b3b27bd29 in 290ms, sequenceid=153, compaction requested=true 2023-04-09 09:53:50,555 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1732612c92cd83e5f8169e1b3b27bd29: 2023-04-09 09:53:50,555 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:53:50,556 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-09 09:53:50,556 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-09 09:53:50,556 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:53:50,556 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:53:50,561 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45588 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-09 09:53:50,562 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): 1732612c92cd83e5f8169e1b3b27bd29/B is initiating minor compaction (all files) 2023-04-09 09:53:50,562 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1732612c92cd83e5f8169e1b3b27bd29/B in TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:50,562 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/504f447e7bb54d5093883d200bb325f5, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/6bba01064e294ec29be6740099bc6773, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/cf656a65e8454cc69f66df09b0316251, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/3d9d91e6b6a04224b63c91c3bcd0e2c1] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp, totalSize=44.5 K 2023-04-09 09:53:50,562 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 57438 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-09 09:53:50,563 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1912): 1732612c92cd83e5f8169e1b3b27bd29/A is initiating minor compaction (all files) 2023-04-09 09:53:50,563 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1732612c92cd83e5f8169e1b3b27bd29/A in TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:50,563 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/f921476f471c463784c584e3a5ecbe41, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/198606809714499ca6fca91e0ce38388, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/42572582c48b444ea0ae4f0fbc26d9c8, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/f1e1fd314ff84c439767c47794cc3b5c] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp, totalSize=56.1 K 2023-04-09 09:53:50,564 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting f921476f471c463784c584e3a5ecbe41, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=92, earliestPutTs=1681034027086 2023-04-09 09:53:50,565 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 198606809714499ca6fca91e0ce38388, keycount=250, bloomtype=ROW, size=16.2 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1681034027724 2023-04-09 09:53:50,565 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 504f447e7bb54d5093883d200bb325f5, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=92, earliestPutTs=1681034027086 2023-04-09 09:53:50,567 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 42572582c48b444ea0ae4f0fbc26d9c8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1681034029109 2023-04-09 09:53:50,567 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 6bba01064e294ec29be6740099bc6773, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1681034027724 2023-04-09 09:53:50,567 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting f1e1fd314ff84c439767c47794cc3b5c, keycount=250, bloomtype=ROW, size=16.4 K, encoding=NONE, compression=NONE, seqNum=153, earliestPutTs=1681034030256 2023-04-09 09:53:50,568 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting cf656a65e8454cc69f66df09b0316251, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1681034029109 2023-04-09 09:53:50,569 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:53:50,570 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 3d9d91e6b6a04224b63c91c3bcd0e2c1, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=153, earliestPutTs=1681034030258 2023-04-09 09:53:50,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:53:50,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:53:50,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:53:50,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:53:50,584 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:53:50,584 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:50,584 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:53:50,584 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:53:50,584 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:50,584 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:50,587 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:50,587 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:50,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:50,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:50,587 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:50,587 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:53:50,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:50,588 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:50,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:53:50,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:50,589 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:53:50,589 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:53:50,589 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:53:50,589 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:53:50,589 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:50,589 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:53:50,590 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:53:50,590 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:53:50,590 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:50,590 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. started... 2023-04-09 09:53:50,590 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1732612c92cd83e5f8169e1b3b27bd29 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-09 09:53:50,590 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=A 2023-04-09 09:53:50,591 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:53:50,591 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=B 2023-04-09 09:53:50,591 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:53:50,591 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=C 2023-04-09 09:53:50,591 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:53:50,591 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): 1732612c92cd83e5f8169e1b3b27bd29#B#compaction#24 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:53:50,613 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1732612c92cd83e5f8169e1b3b27bd29#A#compaction#25 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:53:50,679 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/B/f02c5db68e234ff88bed1f1ba5cc318d as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/f02c5db68e234ff88bed1f1ba5cc318d 2023-04-09 09:53:50,679 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=190 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/A/e1bc9d9021fc40d3890429cf14316478 2023-04-09 09:53:50,693 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1732612c92cd83e5f8169e1b3b27bd29/B of 1732612c92cd83e5f8169e1b3b27bd29 into f02c5db68e234ff88bed1f1ba5cc318d(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:53:50,693 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1732612c92cd83e5f8169e1b3b27bd29: 2023-04-09 09:53:50,693 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29., storeName=1732612c92cd83e5f8169e1b3b27bd29/B, priority=12, startTime=1681034030556; duration=0sec 2023-04-09 09:53:50,693 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:53:50,693 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-09 09:53:50,695 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45588 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-09 09:53:50,695 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): 1732612c92cd83e5f8169e1b3b27bd29/C is initiating minor compaction (all files) 2023-04-09 09:53:50,695 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1732612c92cd83e5f8169e1b3b27bd29/C in TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:50,696 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/0b4aef4af45c4703becfe47daf81bdc9, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/d4962c47376c4769848588a5fbc9171c, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/822298757f664773aab2cf90d9e91f83, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/da1b99f29e1f4d5086a1fcb7f42ce6f0] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp, totalSize=44.5 K 2023-04-09 09:53:50,696 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 0b4aef4af45c4703becfe47daf81bdc9, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=92, earliestPutTs=1681034027086 2023-04-09 09:53:50,697 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting d4962c47376c4769848588a5fbc9171c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1681034027724 2023-04-09 09:53:50,698 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 822298757f664773aab2cf90d9e91f83, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1681034029109 2023-04-09 09:53:50,699 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting da1b99f29e1f4d5086a1fcb7f42ce6f0, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=153, earliestPutTs=1681034030258 2023-04-09 09:53:50,736 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. as already flushing 2023-04-09 09:53:50,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 1732612c92cd83e5f8169e1b3b27bd29 2023-04-09 09:53:50,743 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): 1732612c92cd83e5f8169e1b3b27bd29#C#compaction#28 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:53:50,762 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:50,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39920 deadline: 1681034090758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:50,763 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:50,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39954 deadline: 1681034090758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:50,764 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:50,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39864 deadline: 1681034090758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:50,867 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:50,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39920 deadline: 1681034090864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:50,867 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:50,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39954 deadline: 1681034090865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:50,868 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:50,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39864 deadline: 1681034090865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:51,074 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:51,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39864 deadline: 1681034091069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:51,075 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:51,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39954 deadline: 1681034091070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:51,075 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:51,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39920 deadline: 1681034091070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:51,089 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/A/eac6f83f3f9b415885e9347e0212bf22 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/eac6f83f3f9b415885e9347e0212bf22 2023-04-09 09:53:51,100 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1732612c92cd83e5f8169e1b3b27bd29/A of 1732612c92cd83e5f8169e1b3b27bd29 into eac6f83f3f9b415885e9347e0212bf22(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:53:51,100 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1732612c92cd83e5f8169e1b3b27bd29: 2023-04-09 09:53:51,100 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29., storeName=1732612c92cd83e5f8169e1b3b27bd29/A, priority=12, startTime=1681034030555; duration=0sec 2023-04-09 09:53:51,100 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:53:51,148 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=190 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/B/cc558017a2384ea9b2cf5471581b116b 2023-04-09 09:53:51,178 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/C/9cd565f14d5a47d0b9109c140dba6496 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/9cd565f14d5a47d0b9109c140dba6496 2023-04-09 09:53:51,188 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1732612c92cd83e5f8169e1b3b27bd29/C of 1732612c92cd83e5f8169e1b3b27bd29 into 9cd565f14d5a47d0b9109c140dba6496(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:53:51,188 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1732612c92cd83e5f8169e1b3b27bd29: 2023-04-09 09:53:51,188 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29., storeName=1732612c92cd83e5f8169e1b3b27bd29/C, priority=12, startTime=1681034030556; duration=0sec 2023-04-09 09:53:51,188 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:53:51,380 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:51,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39864 deadline: 1681034091377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:51,380 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:51,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39954 deadline: 1681034091377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:51,381 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:51,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39920 deadline: 1681034091377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:51,610 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=190 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/C/ff5552e27d684f039711c65c622df4cc 2023-04-09 09:53:51,618 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/A/e1bc9d9021fc40d3890429cf14316478 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/e1bc9d9021fc40d3890429cf14316478 2023-04-09 09:53:51,628 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/e1bc9d9021fc40d3890429cf14316478, entries=150, sequenceid=190, filesize=11.8 K 2023-04-09 09:53:51,630 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/B/cc558017a2384ea9b2cf5471581b116b as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/cc558017a2384ea9b2cf5471581b116b 2023-04-09 09:53:51,641 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/cc558017a2384ea9b2cf5471581b116b, entries=150, sequenceid=190, filesize=11.8 K 2023-04-09 09:53:51,645 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/C/ff5552e27d684f039711c65c622df4cc as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/ff5552e27d684f039711c65c622df4cc 2023-04-09 09:53:51,654 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/ff5552e27d684f039711c65c622df4cc, entries=150, sequenceid=190, filesize=11.8 K 2023-04-09 09:53:51,657 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=53.67 KB/54960 for 1732612c92cd83e5f8169e1b3b27bd29 in 1067ms, sequenceid=190, compaction requested=false 2023-04-09 09:53:51,657 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1732612c92cd83e5f8169e1b3b27bd29: 2023-04-09 09:53:51,657 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:51,657 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:53:51,658 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:53:51,658 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:51,658 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:53:51,658 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:53:51,661 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:51,661 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:51,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:51,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:51,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:51,661 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:51,661 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:53:51,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:51,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:51,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:51,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:51,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:51,663 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:53:51,663 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6177d06c[Count = 0] remaining members to acquire global barrier 2023-04-09 09:53:51,663 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:53:51,663 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:51,664 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:51,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:51,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:51,665 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:53:51,665 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:53:51,665 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:53:51,665 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:51,665 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:53:51,666 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:53:51,666 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:51,666 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:51,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:51,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:51,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:51,666 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:53:51,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:51,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:51,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:51,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:51,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:51,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:51,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:51,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:53:51,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:53:51,676 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:53:51,676 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:53:51,676 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:53:51,676 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:53:51,676 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:53:51,679 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:51,679 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:51,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:51,679 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:51,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:51,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:51,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:51,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:51,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:51,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:51,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:51,680 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:51,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:51,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:51,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:51,681 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:51,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:51,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:51,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:51,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:51,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:51,685 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:51,685 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:51,685 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:51,685 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:51,685 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:51,685 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:51,685 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:51,685 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:51,685 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:51,685 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:51,685 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:53:51,685 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:53:51,685 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:53:51,686 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:53:51,686 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:53:51,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:51,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:51,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:51,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:51,786 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:53:51,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:53:51,789 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:53:51,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:53:51,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:53:51,790 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:53:51,790 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:51,790 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:53:51,790 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:53:51,791 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:51,791 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:51,792 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:51,792 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:51,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:51,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:51,793 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:51,793 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:53:51,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:51,793 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:51,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:53:51,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:51,794 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:53:51,794 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:53:51,794 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:53:51,794 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:53:51,794 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:51,794 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:53:51,794 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:53:51,794 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:53:51,794 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:51,795 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. started... 2023-04-09 09:53:51,795 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1732612c92cd83e5f8169e1b3b27bd29 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-09 09:53:51,795 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=A 2023-04-09 09:53:51,795 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:53:51,795 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=B 2023-04-09 09:53:51,795 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:53:51,795 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=C 2023-04-09 09:53:51,795 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:53:51,842 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=204 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/A/2b33909d70fd4427a0e4e0e231f7c2ea 2023-04-09 09:53:51,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 1732612c92cd83e5f8169e1b3b27bd29 2023-04-09 09:53:51,885 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. as already flushing 2023-04-09 09:53:51,912 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:51,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39920 deadline: 1681034091910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:51,913 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:51,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39954 deadline: 1681034091911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:51,913 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681034031899, max timestamp=1681034031911Num uniques 859; 2023-04-09 09:53:51,913 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-09 09:53:51,915 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681034031899, max timestamp=1681034031911Num uniques 859; 2023-04-09 09:53:51,915 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-09 09:53:51,918 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681034031899, max timestamp=1681034031911Num uniques 859; 2023-04-09 09:53:51,919 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-09 09:53:51,965 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:51,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39864 deadline: 1681034091964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:52,018 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:52,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39920 deadline: 1681034092014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:52,019 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:52,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39954 deadline: 1681034092014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:52,069 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:52,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39864 deadline: 1681034092067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:52,224 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:52,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39954 deadline: 1681034092220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:52,225 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:52,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39920 deadline: 1681034092221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:52,275 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:52,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39864 deadline: 1681034092272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:52,276 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=204 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/B/e4b770bdf6eb4b1387d3eec5bf6c253b 2023-04-09 09:53:52,310 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=204 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/C/6d643113098946c38e38f2ccc4c9ebd0 2023-04-09 09:53:52,321 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/A/2b33909d70fd4427a0e4e0e231f7c2ea as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/2b33909d70fd4427a0e4e0e231f7c2ea 2023-04-09 09:53:52,330 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/2b33909d70fd4427a0e4e0e231f7c2ea, entries=150, sequenceid=204, filesize=11.8 K 2023-04-09 09:53:52,333 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/B/e4b770bdf6eb4b1387d3eec5bf6c253b as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/e4b770bdf6eb4b1387d3eec5bf6c253b 2023-04-09 09:53:52,341 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/e4b770bdf6eb4b1387d3eec5bf6c253b, entries=150, sequenceid=204, filesize=11.8 K 2023-04-09 09:53:52,344 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/C/6d643113098946c38e38f2ccc4c9ebd0 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/6d643113098946c38e38f2ccc4c9ebd0 2023-04-09 09:53:52,350 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/6d643113098946c38e38f2ccc4c9ebd0, entries=150, sequenceid=204, filesize=11.8 K 2023-04-09 09:53:52,352 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 1732612c92cd83e5f8169e1b3b27bd29 in 557ms, sequenceid=204, compaction requested=true 2023-04-09 09:53:52,352 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1732612c92cd83e5f8169e1b3b27bd29: 2023-04-09 09:53:52,353 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:52,353 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:53:52,353 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:53:52,353 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:52,354 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:53:52,354 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:53:52,355 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:52,355 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:52,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:52,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:52,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:52,356 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:52,356 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:53:52,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:52,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:52,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:52,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:52,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:52,358 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:53:52,358 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@497a5e39[Count = 0] remaining members to acquire global barrier 2023-04-09 09:53:52,358 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:53:52,358 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:52,359 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:52,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:52,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:52,359 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:53:52,359 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:53:52,359 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:53:52,360 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:52,360 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:53:52,363 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:53:52,363 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:52,363 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:52,363 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:53:52,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:52,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:52,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:52,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:52,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:52,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:52,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:52,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:52,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:52,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:52,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:53:52,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:53:52,366 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:53:52,366 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:53:52,366 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:53:52,366 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:53:52,367 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:53:52,368 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:52,368 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:52,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:52,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:52,368 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:52,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:52,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:52,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:52,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:52,369 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:52,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:52,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:52,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:52,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:52,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:52,370 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:52,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:52,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:52,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:52,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:52,373 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:52,373 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:52,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:52,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:52,373 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:52,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:53:52,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:53:52,374 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:53:52,374 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:53:52,374 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:53:52,374 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:52,374 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:52,374 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:52,374 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:52,374 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:52,374 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:52,373 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:52,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:52,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:52,474 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:53:52,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:53:52,478 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:53:52,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:53:52,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:53:52,480 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:53:52,480 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:52,480 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:53:52,480 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:53:52,480 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:52,480 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:52,482 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:52,482 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:52,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:52,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:52,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:52,483 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:52,483 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:53:52,484 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:52,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:53:52,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:52,484 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:53:52,500 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:53:52,500 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:53:52,500 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:53:52,500 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:52,500 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:53:52,501 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:53:52,501 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:53:52,501 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:52,501 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. started... 2023-04-09 09:53:52,501 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1732612c92cd83e5f8169e1b3b27bd29 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-04-09 09:53:52,501 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=A 2023-04-09 09:53:52,502 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:53:52,502 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=B 2023-04-09 09:53:52,504 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:53:52,504 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=C 2023-04-09 09:53:52,504 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:53:52,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 1732612c92cd83e5f8169e1b3b27bd29 2023-04-09 09:53:52,537 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. as already flushing 2023-04-09 09:53:52,558 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:52,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39954 deadline: 1681034092556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:52,558 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:52,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39920 deadline: 1681034092556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:52,580 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:52,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39864 deadline: 1681034092579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:52,661 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:52,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39954 deadline: 1681034092660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:52,662 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:52,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39920 deadline: 1681034092661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:52,866 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:52,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39954 deadline: 1681034092863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:52,867 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:52,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39920 deadline: 1681034092864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:52,934 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=238 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/A/039f4905f9c946cf9e2e2b12b56d1f53 2023-04-09 09:53:52,968 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=238 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/B/31d61467886642ca88348c46d1bc1902 2023-04-09 09:53:52,989 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=238 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/C/28c9b45329bc486ebdcdb42246ed1000 2023-04-09 09:53:52,997 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/A/039f4905f9c946cf9e2e2b12b56d1f53 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/039f4905f9c946cf9e2e2b12b56d1f53 2023-04-09 09:53:53,007 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/039f4905f9c946cf9e2e2b12b56d1f53, entries=150, sequenceid=238, filesize=11.8 K 2023-04-09 09:53:53,011 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/B/31d61467886642ca88348c46d1bc1902 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/31d61467886642ca88348c46d1bc1902 2023-04-09 09:53:53,019 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/31d61467886642ca88348c46d1bc1902, entries=150, sequenceid=238, filesize=11.8 K 2023-04-09 09:53:53,020 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/C/28c9b45329bc486ebdcdb42246ed1000 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/28c9b45329bc486ebdcdb42246ed1000 2023-04-09 09:53:53,029 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/28c9b45329bc486ebdcdb42246ed1000, entries=150, sequenceid=238, filesize=11.8 K 2023-04-09 09:53:53,030 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=53.67 KB/54960 for 1732612c92cd83e5f8169e1b3b27bd29 in 529ms, sequenceid=238, compaction requested=true 2023-04-09 09:53:53,030 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1732612c92cd83e5f8169e1b3b27bd29: 2023-04-09 09:53:53,030 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:53,030 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:53:53,031 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:53:53,031 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,031 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:53:53,032 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:53:53,034 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,034 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:53,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:53,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:53,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:53,035 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:53,035 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:53:53,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:53,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:53,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:53,040 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:53:53,040 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4db8bb8e[Count = 0] remaining members to acquire global barrier 2023-04-09 09:53:53,040 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:53:53,040 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:53,042 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:53,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:53,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:53,042 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:53:53,042 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:53:53,042 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:53:53,042 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,042 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:53:53,044 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:53:53,044 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,044 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:53,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:53,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:53,044 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:53:53,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:53,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:53,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:53,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:53,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:53,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:53:53,049 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:53:53,049 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:53:53,049 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:53:53,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:53:53,049 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:53:53,049 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:53:53,051 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:53,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:53,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:53,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:53,051 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:53,052 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:53,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:53,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:53,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:53,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:53,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:53,052 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:53,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:53,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:53,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:53,054 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:53,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:53,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,060 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,060 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:53,060 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:53,060 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,060 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:53,060 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:53,060 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:53,060 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:53,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:53,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:53:53,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:53:53,060 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:53,060 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:53:53,061 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:53:53,060 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:53,061 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:53:53,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:53,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:53,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:53,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 1732612c92cd83e5f8169e1b3b27bd29 2023-04-09 09:53:53,090 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1732612c92cd83e5f8169e1b3b27bd29 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-09 09:53:53,090 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=A 2023-04-09 09:53:53,091 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:53:53,091 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=B 2023-04-09 09:53:53,091 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:53:53,091 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=C 2023-04-09 09:53:53,091 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:53:53,110 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=250 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/A/01d62b04516b46998d8f1566f5d640cb 2023-04-09 09:53:53,142 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=250 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/B/9821d2ce7fd7400ebe1afc64dcb5d62a 2023-04-09 09:53:53,161 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:53:53,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:53:53,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:53:53,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:53:53,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:53:53,177 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:53:53,177 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:53,177 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:53:53,177 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:53:53,178 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:53,178 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:53,178 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034033157, max timestamp=1681034033176Num uniques 859; 2023-04-09 09:53:53,178 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:53:53,179 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:53,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:53,179 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:53,179 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034033157, max timestamp=1681034033176Num uniques 859; 2023-04-09 09:53:53,179 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:53:53,180 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,180 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:53:53,180 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034033157, max timestamp=1681034033176Num uniques 859; 2023-04-09 09:53:53,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:53,180 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:53:53,180 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:53,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:53:53,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:53,181 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:53:53,181 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:53:53,181 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:53:53,181 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:53:53,181 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:53,181 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:53:53,181 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:53:53,181 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:53:53,182 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=250 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/C/6c7421348d3f4b69b77f41d770d1ad43 2023-04-09 09:53:53,182 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:53,183 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. started... 2023-04-09 09:53:53,183 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. as already flushing 2023-04-09 09:53:53,183 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:53,183 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:53:53,183 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:53:53,183 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,183 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:53:53,183 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:53:53,185 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:53,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:53,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:53,186 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:53,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:53,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:53,187 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:53,187 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:53:53,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:53,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:53:53,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3203f3ab[Count = 0] remaining members to acquire global barrier 2023-04-09 09:53:53,188 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:53:53,188 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:53,190 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:53,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:53,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:53,190 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:53:53,190 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:53:53,190 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:53:53,191 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,191 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/A/01d62b04516b46998d8f1566f5d640cb as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/01d62b04516b46998d8f1566f5d640cb 2023-04-09 09:53:53,191 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:53:53,194 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:53,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:53,194 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:53:53,194 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:53,194 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:53:53,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:53,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:53,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:53,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:53,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:53,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:53:53,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:53:53,199 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:53:53,199 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:53:53,199 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:53:53,199 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:53:53,199 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:53:53,201 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/01d62b04516b46998d8f1566f5d640cb, entries=150, sequenceid=250, filesize=11.8 K 2023-04-09 09:53:53,202 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/B/9821d2ce7fd7400ebe1afc64dcb5d62a as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/9821d2ce7fd7400ebe1afc64dcb5d62a 2023-04-09 09:53:53,202 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:53,202 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:53,203 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:53,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:53,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:53,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:53,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:53,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:53,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:53,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:53,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:53,204 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:53,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:53,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:53,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:53,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,209 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:53,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:53,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,212 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/9821d2ce7fd7400ebe1afc64dcb5d62a, entries=100, sequenceid=250, filesize=9.4 K 2023-04-09 09:53:53,214 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/C/6c7421348d3f4b69b77f41d770d1ad43 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/6c7421348d3f4b69b77f41d770d1ad43 2023-04-09 09:53:53,215 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:53,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:53,215 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:53,215 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:53:53,215 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:53:53,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:53:53,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:53,215 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:53,215 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:53,215 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:53,216 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39864 deadline: 1681034093209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,216 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:53:53,216 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:53:53,216 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:53,216 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:53,216 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:53,216 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:53,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:53,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:53,225 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/6c7421348d3f4b69b77f41d770d1ad43, entries=100, sequenceid=250, filesize=9.4 K 2023-04-09 09:53:53,226 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=127.47 KB/130530 for 1732612c92cd83e5f8169e1b3b27bd29 in 136ms, sequenceid=250, compaction requested=true 2023-04-09 09:53:53,226 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1732612c92cd83e5f8169e1b3b27bd29: 2023-04-09 09:53:53,227 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:53:53,227 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-09 09:53:53,227 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:53:53,227 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-09 09:53:53,231 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-09 09:53:53,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 1732612c92cd83e5f8169e1b3b27bd29 2023-04-09 09:53:53,234 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1732612c92cd83e5f8169e1b3b27bd29 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-09 09:53:53,236 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681034033207, max timestamp=1681034033233Num uniques 1018; 2023-04-09 09:53:53,236 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681034033207, max timestamp=1681034033233Num uniques 1018; 2023-04-09 09:53:53,236 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-09 09:53:53,236 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-09 09:53:53,237 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=A 2023-04-09 09:53:53,237 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:53:53,238 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=B 2023-04-09 09:53:53,238 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:53:53,238 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=C 2023-04-09 09:53:53,238 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 58060 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-09 09:53:53,238 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-09 09:53:53,238 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): 1732612c92cd83e5f8169e1b3b27bd29/B is initiating minor compaction (all files) 2023-04-09 09:53:53,238 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1732612c92cd83e5f8169e1b3b27bd29/B in TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:53,238 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/f02c5db68e234ff88bed1f1ba5cc318d, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/cc558017a2384ea9b2cf5471581b116b, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/e4b770bdf6eb4b1387d3eec5bf6c253b, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/31d61467886642ca88348c46d1bc1902, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/9821d2ce7fd7400ebe1afc64dcb5d62a] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp, totalSize=56.7 K 2023-04-09 09:53:53,238 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60450 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-09 09:53:53,239 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1912): 1732612c92cd83e5f8169e1b3b27bd29/A is initiating minor compaction (all files) 2023-04-09 09:53:53,239 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1732612c92cd83e5f8169e1b3b27bd29/A in TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:53,239 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/eac6f83f3f9b415885e9347e0212bf22, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/e1bc9d9021fc40d3890429cf14316478, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/2b33909d70fd4427a0e4e0e231f7c2ea, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/039f4905f9c946cf9e2e2b12b56d1f53, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/01d62b04516b46998d8f1566f5d640cb] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp, totalSize=59.0 K 2023-04-09 09:53:53,239 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting f02c5db68e234ff88bed1f1ba5cc318d, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=153, earliestPutTs=1681034029121 2023-04-09 09:53:53,241 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting eac6f83f3f9b415885e9347e0212bf22, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=153, earliestPutTs=1681034029121 2023-04-09 09:53:53,242 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting cc558017a2384ea9b2cf5471581b116b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=190, earliestPutTs=1681034030321 2023-04-09 09:53:53,242 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting e1bc9d9021fc40d3890429cf14316478, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=190, earliestPutTs=1681034030321 2023-04-09 09:53:53,243 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 2b33909d70fd4427a0e4e0e231f7c2ea, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=204, earliestPutTs=1681034030747 2023-04-09 09:53:53,243 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting e4b770bdf6eb4b1387d3eec5bf6c253b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=204, earliestPutTs=1681034030747 2023-04-09 09:53:53,244 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 039f4905f9c946cf9e2e2b12b56d1f53, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=238, earliestPutTs=1681034031936 2023-04-09 09:53:53,244 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 31d61467886642ca88348c46d1bc1902, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=238, earliestPutTs=1681034031936 2023-04-09 09:53:53,244 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 01d62b04516b46998d8f1566f5d640cb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=250, earliestPutTs=1681034032554 2023-04-09 09:53:53,245 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 9821d2ce7fd7400ebe1afc64dcb5d62a, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=250, earliestPutTs=1681034032554 2023-04-09 09:53:53,255 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=288 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/A/bb61906d5fa84fb58aade79bdeca85fa 2023-04-09 09:53:53,265 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): 1732612c92cd83e5f8169e1b3b27bd29#B#compaction#40 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:53:53,280 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1732612c92cd83e5f8169e1b3b27bd29#A#compaction#42 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:53:53,294 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=288 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/B/ffd007ef7a0a480fa314b6b887e75d57 2023-04-09 09:53:53,301 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:53,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39954 deadline: 1681034093299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,302 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:53,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39920 deadline: 1681034093302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,304 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/B/bb99ae491f6b4deebd0913066acea7ef as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/bb99ae491f6b4deebd0913066acea7ef 2023-04-09 09:53:53,306 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/A/6a62561777a64b4fbce46ee93ae54e21 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/6a62561777a64b4fbce46ee93ae54e21 2023-04-09 09:53:53,313 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 1732612c92cd83e5f8169e1b3b27bd29/B of 1732612c92cd83e5f8169e1b3b27bd29 into bb99ae491f6b4deebd0913066acea7ef(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:53:53,313 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1732612c92cd83e5f8169e1b3b27bd29: 2023-04-09 09:53:53,313 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29., storeName=1732612c92cd83e5f8169e1b3b27bd29/B, priority=11, startTime=1681034033227; duration=0sec 2023-04-09 09:53:53,313 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:53:53,313 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-09 09:53:53,317 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:53:53,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:53:53,320 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 1732612c92cd83e5f8169e1b3b27bd29/A of 1732612c92cd83e5f8169e1b3b27bd29 into 6a62561777a64b4fbce46ee93ae54e21(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:53:53,320 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 58060 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-09 09:53:53,320 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1732612c92cd83e5f8169e1b3b27bd29: 2023-04-09 09:53:53,320 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): 1732612c92cd83e5f8169e1b3b27bd29/C is initiating minor compaction (all files) 2023-04-09 09:53:53,320 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29., storeName=1732612c92cd83e5f8169e1b3b27bd29/A, priority=11, startTime=1681034033227; duration=0sec 2023-04-09 09:53:53,320 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1732612c92cd83e5f8169e1b3b27bd29/C in TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:53,320 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:53:53,320 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/9cd565f14d5a47d0b9109c140dba6496, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/ff5552e27d684f039711c65c622df4cc, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/6d643113098946c38e38f2ccc4c9ebd0, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/28c9b45329bc486ebdcdb42246ed1000, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/6c7421348d3f4b69b77f41d770d1ad43] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp, totalSize=56.7 K 2023-04-09 09:53:53,321 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=288 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/C/7a72d5983b58448fa0a3cca630c1275f 2023-04-09 09:53:53,321 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 9cd565f14d5a47d0b9109c140dba6496, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=153, earliestPutTs=1681034029121 2023-04-09 09:53:53,325 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting ff5552e27d684f039711c65c622df4cc, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=190, earliestPutTs=1681034030321 2023-04-09 09:53:53,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:53:53,325 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 6d643113098946c38e38f2ccc4c9ebd0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=204, earliestPutTs=1681034030747 2023-04-09 09:53:53,326 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 28c9b45329bc486ebdcdb42246ed1000, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=238, earliestPutTs=1681034031936 2023-04-09 09:53:53,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:53:53,327 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 6c7421348d3f4b69b77f41d770d1ad43, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=250, earliestPutTs=1681034032554 2023-04-09 09:53:53,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:53:53,327 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:53:53,327 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:53,327 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:53:53,327 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:53:53,328 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:53,328 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:53,329 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:53,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39864 deadline: 1681034093320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,330 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:53,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:53,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:53,330 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:53,331 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,331 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:53:53,332 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:53,333 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/A/bb61906d5fa84fb58aade79bdeca85fa as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/bb61906d5fa84fb58aade79bdeca85fa 2023-04-09 09:53:53,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:53:53,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:53,333 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:53:53,334 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:53:53,334 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:53:53,334 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:53:53,334 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:53,335 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:53:53,339 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/bb61906d5fa84fb58aade79bdeca85fa, entries=200, sequenceid=288, filesize=14.3 K 2023-04-09 09:53:53,340 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/B/ffd007ef7a0a480fa314b6b887e75d57 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/ffd007ef7a0a480fa314b6b887e75d57 2023-04-09 09:53:53,342 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:53:53,342 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:53:53,342 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:53,342 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. started... 2023-04-09 09:53:53,342 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. as already flushing 2023-04-09 09:53:53,342 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:53,342 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:53:53,342 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:53:53,342 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,342 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:53:53,343 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:53:53,344 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): 1732612c92cd83e5f8169e1b3b27bd29#C#compaction#44 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:53:53,345 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,345 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:53,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:53,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:53,346 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:53,346 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:53:53,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:53,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:53,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:53,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:53,352 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:53:53,352 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6da8a137[Count = 0] remaining members to acquire global barrier 2023-04-09 09:53:53,352 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:53:53,352 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:53,352 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/ffd007ef7a0a480fa314b6b887e75d57, entries=150, sequenceid=288, filesize=11.9 K 2023-04-09 09:53:53,353 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:53,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:53,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:53,354 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:53:53,354 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:53:53,354 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:53:53,354 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,354 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:53:53,354 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/C/7a72d5983b58448fa0a3cca630c1275f as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/7a72d5983b58448fa0a3cca630c1275f 2023-04-09 09:53:53,355 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:53:53,356 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,356 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:53,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:53,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:53,356 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:53:53,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:53,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:53,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:53,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:53,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:53,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:53:53,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:53:53,359 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:53:53,359 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:53:53,359 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:53:53,359 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:53:53,359 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:53:53,360 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/7a72d5983b58448fa0a3cca630c1275f, entries=150, sequenceid=288, filesize=11.9 K 2023-04-09 09:53:53,361 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:53,361 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:53,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:53,361 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:53,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:53,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:53,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:53,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:53,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:53,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:53,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:53,389 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,391 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~69.33 KB/70990, heapSize ~188.91 KB/193440, currentSize=114.05 KB/116790 for 1732612c92cd83e5f8169e1b3b27bd29 in 157ms, sequenceid=288, compaction requested=false 2023-04-09 09:53:53,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:53,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:53,391 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1732612c92cd83e5f8169e1b3b27bd29: 2023-04-09 09:53:53,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:53,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:53,395 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:53,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:53,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,404 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:53,404 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:53,404 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,404 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:53,404 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:53,404 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:53,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:53,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:53,416 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034033276, max timestamp=1681034033413Num uniques 859; 2023-04-09 09:53:53,422 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:53:53,416 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1732612c92cd83e5f8169e1b3b27bd29 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-09 09:53:53,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 1732612c92cd83e5f8169e1b3b27bd29 2023-04-09 09:53:53,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:53,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:53,423 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=A 2023-04-09 09:53:53,415 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034033276, max timestamp=1681034033413Num uniques 859; 2023-04-09 09:53:53,423 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:53:53,415 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034033276, max timestamp=1681034033413Num uniques 859; 2023-04-09 09:53:53,412 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/C/526fb8876b8f4b7e8b5041680767e823 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/526fb8876b8f4b7e8b5041680767e823 2023-04-09 09:53:53,410 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:53:53,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:53:53,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:53:53,404 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,424 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:53,424 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:53,424 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:53,424 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:53:53,424 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:53:53,423 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:53:53,424 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:53:53,424 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=B 2023-04-09 09:53:53,424 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:53:53,424 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=C 2023-04-09 09:53:53,424 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:53:53,436 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 1732612c92cd83e5f8169e1b3b27bd29/C of 1732612c92cd83e5f8169e1b3b27bd29 into 526fb8876b8f4b7e8b5041680767e823(size=12.2 K), total size for store is 24.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:53:53,436 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1732612c92cd83e5f8169e1b3b27bd29: 2023-04-09 09:53:53,436 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29., storeName=1732612c92cd83e5f8169e1b3b27bd29/C, priority=11, startTime=1681034033227; duration=0sec 2023-04-09 09:53:53,436 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:53:53,441 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=312 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/A/c70ec227af05431e8bc8bb661ba6c13a 2023-04-09 09:53:53,478 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681034033449, max timestamp=1681034033476Num uniques 859; 2023-04-09 09:53:53,478 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681034033449, max timestamp=1681034033476Num uniques 859; 2023-04-09 09:53:53,478 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-09 09:53:53,478 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681034033449, max timestamp=1681034033476Num uniques 859; 2023-04-09 09:53:53,478 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-09 09:53:53,478 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-09 09:53:53,512 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:53,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39954 deadline: 1681034093511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,513 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681034033493, max timestamp=1681034033511Num uniques 1118; 2023-04-09 09:53:53,513 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681034033493, max timestamp=1681034033511Num uniques 1118; 2023-04-09 09:53:53,513 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=7688, suffix heap size=144240,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=168 2023-04-09 09:53:53,514 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=7688, suffix heap size=144240,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=168 2023-04-09 09:53:53,514 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681034033493, max timestamp=1681034033511Num uniques 1118; 2023-04-09 09:53:53,514 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=7688, suffix heap size=144240,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=168 2023-04-09 09:53:53,524 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:53:53,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:53:53,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:53:53,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:53:53,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:53:53,528 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:53:53,528 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:53,528 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:53:53,528 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:53:53,529 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:53,529 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:53,531 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,531 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:53,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:53,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:53,532 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,532 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:53:53,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:53,532 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:53,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:53:53,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:53,533 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:53:53,536 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:53:53,536 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:53:53,536 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:53,536 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:53:53,539 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:53:53,539 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:53:53,539 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:53,539 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. started... 2023-04-09 09:53:53,540 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. as already flushing 2023-04-09 09:53:53,540 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:53,540 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:53:53,540 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:53:53,540 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,540 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:53:53,540 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:53:53,542 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,542 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:53,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:53,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:53,542 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:53,542 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:53:53,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:53,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:53,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:53,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:53,544 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:53:53,545 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76b8a12d[Count = 0] remaining members to acquire global barrier 2023-04-09 09:53:53,545 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:53:53,545 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:53,546 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:53,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:53,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:53,546 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:53:53,546 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:53:53,546 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:53:53,548 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,548 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:53:53,550 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:53:53,550 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:53,550 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:53:53,550 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:53,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:53,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:53,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:53,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:53,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:53,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:53,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:53:53,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:53:53,560 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:53:53,560 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:53:53,560 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:53:53,560 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:53:53,560 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:53:53,561 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:53,562 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:53,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:53,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:53,562 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:53,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:53,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:53,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:53,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:53,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:53,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:53,563 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:53,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:53,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:53,566 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:53,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:53,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:53,570 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:53,570 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:53,570 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:53,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:53,570 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:53,571 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:53,571 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,571 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:53,571 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:53,571 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:53,571 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:53,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:53:53,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:53:53,571 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:53:53,571 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:53:53,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:53,572 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:53:53,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:53,578 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681034033546, max timestamp=1681034033573Num uniques 1027; 2023-04-09 09:53:53,578 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-09 09:53:53,580 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681034033546, max timestamp=1681034033573Num uniques 1027; 2023-04-09 09:53:53,580 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-09 09:53:53,583 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681034033546, max timestamp=1681034033573Num uniques 1027; 2023-04-09 09:53:53,583 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-09 09:53:53,630 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681034033598, max timestamp=1681034033625Num uniques 1086; 2023-04-09 09:53:53,630 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-09 09:53:53,632 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681034033598, max timestamp=1681034033625Num uniques 1086; 2023-04-09 09:53:53,632 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-09 09:53:53,634 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681034033598, max timestamp=1681034033625Num uniques 1086; 2023-04-09 09:53:53,634 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-09 09:53:53,671 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:53:53,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:53:53,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:53:53,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:53:53,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:53:53,692 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:53:53,692 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:53,692 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:53:53,692 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:53:53,692 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:53,692 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:53,700 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:53,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:53,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:53,701 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:53,701 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,701 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:53:53,701 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:53,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:53:53,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:53,702 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:53:53,709 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:53:53,710 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:53:53,710 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:53:53,711 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:53,711 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:53:53,711 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681034033702, max timestamp=1681034033709Num uniques 1095; 2023-04-09 09:53:53,711 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-09 09:53:53,712 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681034033702, max timestamp=1681034033709Num uniques 1095; 2023-04-09 09:53:53,712 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681034033702, max timestamp=1681034033709Num uniques 1095; 2023-04-09 09:53:53,712 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-09 09:53:53,712 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-09 09:53:53,717 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:53:53,717 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:53,717 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. started... 2023-04-09 09:53:53,717 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:53:53,717 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. as already flushing 2023-04-09 09:53:53,717 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:53,717 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:53:53,717 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:53:53,717 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,718 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:53:53,718 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:53:53,726 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:53,727 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:53,727 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:53:53,732 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:53,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:53,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:53,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:53,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:53,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:53,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:53:53,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ed400bf[Count = 0] remaining members to acquire global barrier 2023-04-09 09:53:53,738 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:53:53,738 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:53,739 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:53,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:53,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:53,740 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:53:53,740 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:53:53,740 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:53:53,740 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,740 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:53:53,741 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,741 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:53:53,741 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:53,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:53,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:53,741 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:53:53,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:53,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:53,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:53,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:53,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:53,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:53:53,747 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:53:53,747 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:53:53,747 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:53:53,747 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:53:53,747 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:53:53,747 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:53:53,749 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:53,750 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:53,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:53,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:53,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:53,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:53,750 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:53,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:53,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:53,751 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:53,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:53,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:53,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:53,753 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9334, totalHeapSize=21456, min timestamp=1681034033731, max timestamp=1681034033750Num uniques 1054; 2023-04-09 09:53:53,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:53,754 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48264, new segment data size=9334, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-09 09:53:53,754 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9334, totalHeapSize=21456, min timestamp=1681034033731, max timestamp=1681034033750Num uniques 1054; 2023-04-09 09:53:53,754 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48264, new segment data size=9334, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-09 09:53:53,756 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681034033731, max timestamp=1681034033750Num uniques 1054; 2023-04-09 09:53:53,756 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-09 09:53:53,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:53,766 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:53,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:53,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,770 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,770 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:53,770 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:53,770 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,770 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:53,770 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:53,770 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:53,770 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:53:53,770 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:53,770 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:53,770 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:53,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:53:53,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:53:53,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:53,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:53,771 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:53:53,771 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:53:53,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:53,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:53,797 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1681034033779, max timestamp=1681034033794Num uniques 1063; 2023-04-09 09:53:53,797 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=9748, suffix heap size=138520,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=213 2023-04-09 09:53:53,797 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1681034033779, max timestamp=1681034033794Num uniques 1063; 2023-04-09 09:53:53,797 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=9748, suffix heap size=138520,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=213 2023-04-09 09:53:53,799 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1681034033779, max timestamp=1681034033794Num uniques 1013; 2023-04-09 09:53:53,799 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-09 09:53:53,863 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1681034033836, max timestamp=1681034033859Num uniques 1072; 2023-04-09 09:53:53,863 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=12452, suffix heap size=139456,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=272 2023-04-09 09:53:53,863 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1681034033836, max timestamp=1681034033859Num uniques 1072; 2023-04-09 09:53:53,863 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1681034033836, max timestamp=1681034033859Num uniques 1022; 2023-04-09 09:53:53,863 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=12452, suffix heap size=134256,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=272 2023-04-09 09:53:53,863 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=12452, suffix heap size=139456,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=272 2023-04-09 09:53:53,871 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:53:53,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:53:53,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:53:53,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:53:53,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:53:53,877 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:53:53,877 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:53,877 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:53:53,877 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:53:53,877 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=312 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/B/96a26d29b9e642a88f3156ed5953db8f 2023-04-09 09:53:53,878 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:53,878 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:53,879 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,879 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:53,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:53,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:53,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:53,880 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,880 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:53:53,881 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:53,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:53:53,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:53,882 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:53:53,893 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:53:53,893 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:53:53,893 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:53:53,893 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:53,893 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:53:53,896 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:53:53,896 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:53:53,896 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:53,896 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. started... 2023-04-09 09:53:53,896 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. as already flushing 2023-04-09 09:53:53,896 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:53,896 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:53:53,896 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:53:53,896 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,896 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:53:53,897 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:53:53,899 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,899 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:53,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:53,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:53,899 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:53,899 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:53:53,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:53,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:53,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:53,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:53,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:53:53,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53e73a65[Count = 0] remaining members to acquire global barrier 2023-04-09 09:53:53,902 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:53:53,902 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:53,905 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:53,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:53,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:53,905 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:53:53,905 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,905 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:53:53,905 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:53:53,905 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:53:53,907 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:53:53,907 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:53,907 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,907 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:53:53,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:53,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:53,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:53,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:53,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:53,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:53,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:53,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:53:53,910 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:53:53,910 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:53:53,910 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:53:53,910 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:53:53,910 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:53:53,910 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:53:53,911 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=312 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/C/811704e8e43b41d3a22a3b5fe94ab0eb 2023-04-09 09:53:53,912 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:53,912 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:53,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:53,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:53,912 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:53,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:53,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:53,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:53,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:53,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:53,913 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:53,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:53,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:53,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:53,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:53,915 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:53,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:53,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,918 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,918 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:53,918 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:53,918 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,918 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/A/c70ec227af05431e8bc8bb661ba6c13a as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/c70ec227af05431e8bc8bb661ba6c13a 2023-04-09 09:53:53,918 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:53,918 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:53,918 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:53,918 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:53,918 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:53,918 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:53,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:53:53,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:53:53,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:53,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:53,919 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:53:53,919 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:53:53,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:53,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:53,924 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/c70ec227af05431e8bc8bb661ba6c13a, entries=150, sequenceid=312, filesize=11.9 K 2023-04-09 09:53:53,925 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/B/96a26d29b9e642a88f3156ed5953db8f as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/96a26d29b9e642a88f3156ed5953db8f 2023-04-09 09:53:53,929 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/96a26d29b9e642a88f3156ed5953db8f, entries=150, sequenceid=312, filesize=11.9 K 2023-04-09 09:53:53,930 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/C/811704e8e43b41d3a22a3b5fe94ab0eb as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/811704e8e43b41d3a22a3b5fe94ab0eb 2023-04-09 09:53:53,935 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/811704e8e43b41d3a22a3b5fe94ab0eb, entries=150, sequenceid=312, filesize=11.9 K 2023-04-09 09:53:53,936 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~86.11 KB/88176, currentSize=114.05 KB/116790 for 1732612c92cd83e5f8169e1b3b27bd29 in 521ms, sequenceid=312, compaction requested=true 2023-04-09 09:53:53,937 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1732612c92cd83e5f8169e1b3b27bd29: 2023-04-09 09:53:53,937 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:53:53,937 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-09 09:53:53,937 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-09 09:53:53,937 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:53:53,937 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:53:53,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 1732612c92cd83e5f8169e1b3b27bd29 2023-04-09 09:53:53,941 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1732612c92cd83e5f8169e1b3b27bd29 3/3 column families, dataSize=120.76 KB heapSize=334.55 KB 2023-04-09 09:53:53,941 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=A 2023-04-09 09:53:53,942 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:53:53,942 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=B 2023-04-09 09:53:53,942 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:53:53,942 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=C 2023-04-09 09:53:53,942 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:53:53,943 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39292 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-09 09:53:53,943 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1912): 1732612c92cd83e5f8169e1b3b27bd29/A is initiating minor compaction (all files) 2023-04-09 09:53:53,943 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36852 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-09 09:53:53,943 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1732612c92cd83e5f8169e1b3b27bd29/A in TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:53,943 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): 1732612c92cd83e5f8169e1b3b27bd29/B is initiating minor compaction (all files) 2023-04-09 09:53:53,943 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/6a62561777a64b4fbce46ee93ae54e21, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/bb61906d5fa84fb58aade79bdeca85fa, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/c70ec227af05431e8bc8bb661ba6c13a] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp, totalSize=38.4 K 2023-04-09 09:53:53,943 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1732612c92cd83e5f8169e1b3b27bd29/B in TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:53,943 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/bb99ae491f6b4deebd0913066acea7ef, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/ffd007ef7a0a480fa314b6b887e75d57, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/96a26d29b9e642a88f3156ed5953db8f] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp, totalSize=36.0 K 2023-04-09 09:53:53,944 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 6a62561777a64b4fbce46ee93ae54e21, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=250, earliestPutTs=1681034031936 2023-04-09 09:53:53,948 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting bb99ae491f6b4deebd0913066acea7ef, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=250, earliestPutTs=1681034031936 2023-04-09 09:53:53,949 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting bb61906d5fa84fb58aade79bdeca85fa, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=288, earliestPutTs=1681034033207 2023-04-09 09:53:53,950 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting ffd007ef7a0a480fa314b6b887e75d57, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=288, earliestPutTs=1681034033207 2023-04-09 09:53:53,950 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting c70ec227af05431e8bc8bb661ba6c13a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1681034033276 2023-04-09 09:53:53,951 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 96a26d29b9e642a88f3156ed5953db8f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1681034033276 2023-04-09 09:53:53,970 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1732612c92cd83e5f8169e1b3b27bd29#A#compaction#49 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:53:53,978 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:53,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39954 deadline: 1681034093969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,981 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): 1732612c92cd83e5f8169e1b3b27bd29#B#compaction#50 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:53:53,983 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:53,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39920 deadline: 1681034093972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:53,983 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:53,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39864 deadline: 1681034093974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,019 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:53:54,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:53:54,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:53:54,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:53:54,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:53:54,031 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:53:54,031 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:54,031 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:53:54,031 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:53:54,031 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:54,032 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:54,034 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:54,034 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:54,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:54,035 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,035 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:53:54,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:54,035 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:54,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:53:54,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:54,036 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:53:54,041 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:53:54,041 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:53:54,041 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:53:54,041 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:54,041 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:53:54,048 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:53:54,048 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:53:54,048 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=466 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/A/2457b724b07a4d4baddda6fb98698332 2023-04-09 09:53:54,050 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:54,051 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. started... 2023-04-09 09:53:54,051 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. as already flushing 2023-04-09 09:53:54,053 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:54,053 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:53:54,053 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:53:54,053 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,053 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:53:54,053 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:53:54,055 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,056 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:54,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:54,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:54,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:54,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:54,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:54,057 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:54,057 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:53:54,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,058 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/B/bc215c92b27f4bfcb2182f9fee5eeaa0 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/bc215c92b27f4bfcb2182f9fee5eeaa0 2023-04-09 09:53:54,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:54,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:53:54,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7bee9837[Count = 0] remaining members to acquire global barrier 2023-04-09 09:53:54,058 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:53:54,058 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:54,060 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:54,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:54,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:54,061 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:53:54,061 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:53:54,061 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:53:54,061 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,061 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:53:54,062 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:54,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:54,063 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:53:54,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:54,063 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:54,063 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:53:54,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:54,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:54,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:54,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:54,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:53:54,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:53:54,069 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:53:54,069 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:53:54,069 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:53:54,069 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:53:54,069 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:53:54,073 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1732612c92cd83e5f8169e1b3b27bd29/B of 1732612c92cd83e5f8169e1b3b27bd29 into bc215c92b27f4bfcb2182f9fee5eeaa0(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:53:54,073 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1732612c92cd83e5f8169e1b3b27bd29: 2023-04-09 09:53:54,073 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29., storeName=1732612c92cd83e5f8169e1b3b27bd29/B, priority=13, startTime=1681034033937; duration=0sec 2023-04-09 09:53:54,073 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:53:54,073 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:54,073 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-09 09:53:54,073 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:54,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:54,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:54,073 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:54,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:54,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:54,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:54,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:54,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:54,075 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36852 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-09 09:53:54,075 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): 1732612c92cd83e5f8169e1b3b27bd29/C is initiating minor compaction (all files) 2023-04-09 09:53:54,075 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,075 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1732612c92cd83e5f8169e1b3b27bd29/C in TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:54,075 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/526fb8876b8f4b7e8b5041680767e823, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/7a72d5983b58448fa0a3cca630c1275f, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/811704e8e43b41d3a22a3b5fe94ab0eb] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp, totalSize=36.0 K 2023-04-09 09:53:54,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:54,076 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 526fb8876b8f4b7e8b5041680767e823, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=250, earliestPutTs=1681034031936 2023-04-09 09:53:54,082 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:54,079 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39954 deadline: 1681034094080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:54,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:54,094 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:54,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:54,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39920 deadline: 1681034094085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,095 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:54,095 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 7a72d5983b58448fa0a3cca630c1275f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=288, earliestPutTs=1681034033207 2023-04-09 09:53:54,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39864 deadline: 1681034094086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:54,095 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 811704e8e43b41d3a22a3b5fe94ab0eb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1681034033276 2023-04-09 09:53:54,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:54,098 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:54,098 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:54,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:54,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:54,099 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,099 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:54,099 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:54,099 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:54,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:53:54,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:53:54,100 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:53:54,100 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:53:54,099 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,100 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:54,100 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:54,100 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:54,102 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=466 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/B/fd85c1ef748648379e8c0bbb223a2862 2023-04-09 09:53:54,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:54,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:54,104 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:53:54,113 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): 1732612c92cd83e5f8169e1b3b27bd29#C#compaction#52 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:53:54,141 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=466 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/C/06ac5987d7274e5db28161b21b47152a 2023-04-09 09:53:54,148 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/A/2457b724b07a4d4baddda6fb98698332 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/2457b724b07a4d4baddda6fb98698332 2023-04-09 09:53:54,161 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/2457b724b07a4d4baddda6fb98698332, entries=200, sequenceid=466, filesize=14.3 K 2023-04-09 09:53:54,163 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/B/fd85c1ef748648379e8c0bbb223a2862 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/fd85c1ef748648379e8c0bbb223a2862 2023-04-09 09:53:54,178 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/fd85c1ef748648379e8c0bbb223a2862, entries=150, sequenceid=466, filesize=11.9 K 2023-04-09 09:53:54,192 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/C/06ac5987d7274e5db28161b21b47152a as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/06ac5987d7274e5db28161b21b47152a 2023-04-09 09:53:54,200 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:53:54,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:53:54,202 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/06ac5987d7274e5db28161b21b47152a, entries=150, sequenceid=466, filesize=11.9 K 2023-04-09 09:53:54,204 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~334.50 KB/342528, currentSize=67.09 KB/68700 for 1732612c92cd83e5f8169e1b3b27bd29 in 263ms, sequenceid=466, compaction requested=false 2023-04-09 09:53:54,204 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1732612c92cd83e5f8169e1b3b27bd29: 2023-04-09 09:53:54,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:53:54,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:53:54,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:53:54,206 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:53:54,206 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:54,206 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:53:54,206 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:53:54,207 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:54,207 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:54,208 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:54,208 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:54,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:54,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:54,209 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,209 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:53:54,209 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:54,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:53:54,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:54,210 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:53:54,210 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:53:54,210 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:53:54,210 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:53:54,210 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:54,210 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:53:54,211 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:53:54,211 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:53:54,211 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:54,211 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. started... 2023-04-09 09:53:54,211 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1732612c92cd83e5f8169e1b3b27bd29 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-09 09:53:54,211 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=A 2023-04-09 09:53:54,211 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:53:54,211 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=B 2023-04-09 09:53:54,212 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:53:54,212 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=C 2023-04-09 09:53:54,212 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:53:54,246 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=480 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/A/2f7e0c607cc443908347c3ba6b5420bf 2023-04-09 09:53:54,299 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=480 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/B/c000684a5bd5435da4125fafd3314f0d 2023-04-09 09:53:54,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 1732612c92cd83e5f8169e1b3b27bd29 2023-04-09 09:53:54,308 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. as already flushing 2023-04-09 09:53:54,351 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=480 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/C/53c65dcf65d041caa45cc6c4b0aecf52 2023-04-09 09:53:54,354 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:54,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39920 deadline: 1681034094350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,361 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/A/2f7e0c607cc443908347c3ba6b5420bf as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/2f7e0c607cc443908347c3ba6b5420bf 2023-04-09 09:53:54,364 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:54,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39864 deadline: 1681034094359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,376 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:54,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39954 deadline: 1681034094361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,382 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/2f7e0c607cc443908347c3ba6b5420bf, entries=150, sequenceid=480, filesize=11.9 K 2023-04-09 09:53:54,385 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/B/c000684a5bd5435da4125fafd3314f0d as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/c000684a5bd5435da4125fafd3314f0d 2023-04-09 09:53:54,392 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/c000684a5bd5435da4125fafd3314f0d, entries=150, sequenceid=480, filesize=11.9 K 2023-04-09 09:53:54,394 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/C/53c65dcf65d041caa45cc6c4b0aecf52 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/53c65dcf65d041caa45cc6c4b0aecf52 2023-04-09 09:53:54,414 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/53c65dcf65d041caa45cc6c4b0aecf52, entries=150, sequenceid=480, filesize=11.9 K 2023-04-09 09:53:54,421 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for 1732612c92cd83e5f8169e1b3b27bd29 in 210ms, sequenceid=480, compaction requested=true 2023-04-09 09:53:54,421 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1732612c92cd83e5f8169e1b3b27bd29: 2023-04-09 09:53:54,422 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:54,422 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:53:54,423 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:53:54,423 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,423 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:53:54,423 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:53:54,425 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,425 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:54,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:54,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:54,426 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:54,426 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:53:54,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:54,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:54,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:54,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:54,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:53:54,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48ecefae[Count = 0] remaining members to acquire global barrier 2023-04-09 09:53:54,430 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:53:54,430 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:54,438 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:54,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:54,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:54,438 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:53:54,438 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:53:54,438 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:53:54,439 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,439 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:53:54,444 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/A/75142d4b35234b77b8b69143120ae0cf as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/75142d4b35234b77b8b69143120ae0cf 2023-04-09 09:53:54,445 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,445 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:53:54,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:54,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:54,445 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:54,445 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:53:54,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:54,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:54,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:54,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:54,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:54,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:53:54,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:53:54,453 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:53:54,453 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:53:54,453 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:53:54,453 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:53:54,453 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:53:54,454 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1732612c92cd83e5f8169e1b3b27bd29/A of 1732612c92cd83e5f8169e1b3b27bd29 into 75142d4b35234b77b8b69143120ae0cf(size=12.4 K), total size for store is 38.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:53:54,455 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1732612c92cd83e5f8169e1b3b27bd29: 2023-04-09 09:53:54,455 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29., storeName=1732612c92cd83e5f8169e1b3b27bd29/A, priority=13, startTime=1681034033937; duration=0sec 2023-04-09 09:53:54,455 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:53:54,457 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:54,457 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:54,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:54,457 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:54,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:54,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:54,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:54,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:54,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:54,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:54,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:54,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:54,458 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:54,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:54,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:54,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 1732612c92cd83e5f8169e1b3b27bd29 2023-04-09 09:53:54,460 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1732612c92cd83e5f8169e1b3b27bd29 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-09 09:53:54,460 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=A 2023-04-09 09:53:54,460 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:53:54,460 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=B 2023-04-09 09:53:54,460 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:53:54,461 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=C 2023-04-09 09:53:54,461 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:53:54,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,462 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:54,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:54,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,470 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:54,470 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:54,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:54,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:54,470 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,470 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:54,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:53:54,471 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:53:54,470 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:54,471 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:54,471 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,471 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:54,471 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:54,471 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:54,471 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:53:54,471 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:53:54,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:54,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:54,484 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:54,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39926 deadline: 1681034094480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,484 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:54,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39864 deadline: 1681034094480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,484 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:54,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39920 deadline: 1681034094481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,489 DEBUG [Thread-141] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8252 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29., hostname=jenkins-hbase4.apache.org,42467,1681034019544, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-09 09:53:54,493 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:54,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39954 deadline: 1681034094484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,499 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:54,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39904 deadline: 1681034094499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,508 DEBUG [Thread-143] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8269 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29., hostname=jenkins-hbase4.apache.org,42467,1681034019544, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-09 09:53:54,522 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=502 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/A/bd5328ad650a4d6cace5df36c0be431c 2023-04-09 09:53:54,545 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/C/2b88ce5648544df1accedbf91c43d3b5 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/2b88ce5648544df1accedbf91c43d3b5 2023-04-09 09:53:54,559 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1732612c92cd83e5f8169e1b3b27bd29/C of 1732612c92cd83e5f8169e1b3b27bd29 into 2b88ce5648544df1accedbf91c43d3b5(size=12.4 K), total size for store is 36.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:53:54,559 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1732612c92cd83e5f8169e1b3b27bd29: 2023-04-09 09:53:54,559 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29., storeName=1732612c92cd83e5f8169e1b3b27bd29/C, priority=13, startTime=1681034033937; duration=0sec 2023-04-09 09:53:54,559 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:53:54,571 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:53:54,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:53:54,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:53:54,578 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=502 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/B/793f7129fbc84e67a8d6eb9b658b2db3 2023-04-09 09:53:54,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:53:54,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:53:54,592 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:53:54,592 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:54,592 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:53:54,592 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:53:54,593 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:54,593 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:54,595 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:54,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39920 deadline: 1681034094591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,595 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,595 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:54,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:54,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:54,596 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,596 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:53:54,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:54,597 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:54,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:53:54,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:54,598 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:53:54,601 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:54,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39954 deadline: 1681034094596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,609 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:53:54,610 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:53:54,610 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:53:54,610 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:54,610 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:53:54,612 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:53:54,612 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:53:54,612 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:54,612 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. started... 2023-04-09 09:53:54,612 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. as already flushing 2023-04-09 09:53:54,612 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:54,612 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:53:54,612 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:53:54,612 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,652 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:53:54,652 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:53:54,657 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:54,657 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,657 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:54,657 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:53:54,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:54,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:54,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:54,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:54,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:54,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:54,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:53:54,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16e3dff4[Count = 0] remaining members to acquire global barrier 2023-04-09 09:53:54,662 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:53:54,662 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:54,664 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:54,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:54,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:54,664 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:53:54,664 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:53:54,664 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:53:54,666 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,666 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:53:54,667 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:53:54,667 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:54,667 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,667 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:53:54,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:54,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:54,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:54,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:54,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:54,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:54,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:54,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:53:54,673 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:53:54,673 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:53:54,673 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:53:54,673 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:53:54,673 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:53:54,673 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:53:54,676 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:54,676 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:54,676 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:54,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:54,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:54,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:54,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:54,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:54,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:54,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:54,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:54,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:54,678 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:54,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:54,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:54,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,681 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:54,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:54,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,684 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:54,684 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:54,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:54,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:54,684 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:53:54,685 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:53:54,684 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:54,684 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:54,685 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:54,685 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,685 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:53:54,685 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:53:54,685 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:54,685 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:54,685 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:54,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:54,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:54,693 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:54,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39864 deadline: 1681034094692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,785 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:53:54,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:53:54,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:53:54,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:53:54,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:53:54,789 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:53:54,789 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:54,789 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:53:54,789 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:53:54,790 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:54,790 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:54,795 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,795 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,795 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:53:54,797 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:54,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:54,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:54,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:54,798 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:54,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:53:54,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:54,798 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:53:54,799 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:53:54,799 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:53:54,799 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:53:54,799 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:54,799 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:53:54,799 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:53:54,799 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:53:54,799 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:54,799 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. started... 2023-04-09 09:53:54,799 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. as already flushing 2023-04-09 09:53:54,799 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:54,799 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:53:54,799 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:53:54,800 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,800 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:53:54,800 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:53:54,802 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:54,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39920 deadline: 1681034094797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,803 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:54,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39954 deadline: 1681034094802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,805 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,805 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:54,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:54,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:54,806 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:54,806 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:53:54,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:54,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:54,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:54,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:54,808 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:53:54,808 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c3dfb5f[Count = 0] remaining members to acquire global barrier 2023-04-09 09:53:54,808 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:53:54,808 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:54,811 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:54,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:54,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:54,811 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:53:54,811 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:53:54,811 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:53:54,812 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,812 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:53:54,813 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:53:54,813 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:54,813 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:53:54,814 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:54,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:54,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:54,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:54,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:54,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:54,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:54,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:53:54,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:53:54,817 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:53:54,818 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:53:54,818 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:53:54,818 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:53:54,818 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:53:54,819 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:54,820 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:54,820 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:54,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:54,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:54,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:54,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:54,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:54,820 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:54,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:54,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:54,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:54,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:54,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:54,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:54,822 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:54,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:54,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,826 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,826 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:54,826 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:54,826 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:54,826 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:54,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:54,826 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:54,826 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:54,826 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:54,826 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:54,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:54,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:54,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:53:54,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:53:54,826 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:54,827 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:53:54,827 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:53:54,827 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:53:54,927 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:53:54,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:53:54,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:53:54,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:53:54,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:53:54,931 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:53:54,931 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:54,931 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:53:54,931 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:53:54,932 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:54,932 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:54,934 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,934 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:54,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:54,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:54,934 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,934 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:53:54,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:54,934 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:54,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:53:54,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:54,935 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:53:54,935 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:53:54,935 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:53:54,935 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:53:54,935 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:54,935 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:53:54,935 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:53:54,935 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:53:54,935 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:54,936 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. started... 2023-04-09 09:53:54,936 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. as already flushing 2023-04-09 09:53:54,936 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:54,936 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:53:54,936 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:53:54,936 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,936 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:53:54,936 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:53:54,941 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:54,941 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:54,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:54,941 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:54,941 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:53:54,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:54,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:54,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:54,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:54,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:53:54,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@731debcf[Count = 0] remaining members to acquire global barrier 2023-04-09 09:53:54,944 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:53:54,944 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:54,947 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:54,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:54,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:54,947 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:53:54,947 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:53:54,947 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:53:54,947 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,947 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:53:54,951 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:53:54,951 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,951 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:54,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:54,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:54,951 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:53:54,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:54,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:54,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:54,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:54,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:54,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:53:54,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:53:54,954 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:53:54,954 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:53:54,954 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:53:54,954 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:53:54,954 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:53:54,955 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:54,955 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:54,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:54,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:54,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:54,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:54,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:54,955 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:54,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:54,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:54,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:54,956 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:54,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:54,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:54,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:54,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,957 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:54,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:54,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,959 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,959 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:54,959 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:54,959 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:54,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:54,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:54,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:53:54,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:53:54,959 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:54,959 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:54,960 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:54,960 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:54,960 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:54,960 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:54,960 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:53:54,960 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:53:54,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:54,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:54,996 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:54,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39864 deadline: 1681034094996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,060 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:53:55,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:53:55,062 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=502 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/C/6aeaeb44f4c448148933064a9471b2cd 2023-04-09 09:53:55,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:53:55,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:53:55,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:53:55,064 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:53:55,064 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:55,064 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:53:55,064 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:53:55,065 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:55,065 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:55,066 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,066 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:55,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:55,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:55,066 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,066 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:53:55,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:55,067 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:55,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:53:55,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:55,068 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:53:55,068 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:53:55,068 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:53:55,068 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:53:55,068 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:55,069 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:53:55,069 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/A/bd5328ad650a4d6cace5df36c0be431c as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/bd5328ad650a4d6cace5df36c0be431c 2023-04-09 09:53:55,070 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:53:55,070 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:53:55,070 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:55,070 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. started... 2023-04-09 09:53:55,070 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. as already flushing 2023-04-09 09:53:55,070 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:55,070 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:53:55,070 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:53:55,070 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,070 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:53:55,070 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:53:55,072 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,072 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:55,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:55,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:55,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:55,073 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:55,073 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:53:55,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:55,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:55,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:55,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:53:55,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c23fb9d[Count = 0] remaining members to acquire global barrier 2023-04-09 09:53:55,077 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:53:55,077 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:55,079 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:55,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:55,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:55,080 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,080 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:53:55,080 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:53:55,080 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:53:55,080 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:53:55,081 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:53:55,081 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:55,081 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:53:55,082 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:55,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:55,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:55,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:55,083 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/bd5328ad650a4d6cace5df36c0be431c, entries=200, sequenceid=502, filesize=14.3 K 2023-04-09 09:53:55,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:55,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,085 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/B/793f7129fbc84e67a8d6eb9b658b2db3 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/793f7129fbc84e67a8d6eb9b658b2db3 2023-04-09 09:53:55,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:55,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:55,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:53:55,087 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:53:55,087 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:53:55,087 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:53:55,087 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:53:55,087 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:53:55,087 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:53:55,088 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:55,088 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:55,088 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:55,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:55,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:55,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:55,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:55,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:55,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:55,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:55,090 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:55,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:55,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:55,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:55,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:55,092 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/793f7129fbc84e67a8d6eb9b658b2db3, entries=150, sequenceid=502, filesize=11.9 K 2023-04-09 09:53:55,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,093 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/C/6aeaeb44f4c448148933064a9471b2cd as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/6aeaeb44f4c448148933064a9471b2cd 2023-04-09 09:53:55,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:55,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:55,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,099 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/6aeaeb44f4c448148933064a9471b2cd, entries=150, sequenceid=502, filesize=11.9 K 2023-04-09 09:53:55,101 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for 1732612c92cd83e5f8169e1b3b27bd29 in 641ms, sequenceid=502, compaction requested=true 2023-04-09 09:53:55,101 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1732612c92cd83e5f8169e1b3b27bd29: 2023-04-09 09:53:55,101 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:53:55,101 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-09 09:53:55,101 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-09 09:53:55,101 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:53:55,101 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:53:55,103 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 54170 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-09 09:53:55,103 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1912): 1732612c92cd83e5f8169e1b3b27bd29/A is initiating minor compaction (all files) 2023-04-09 09:53:55,103 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49290 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-09 09:53:55,103 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1732612c92cd83e5f8169e1b3b27bd29/A in TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:55,103 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): 1732612c92cd83e5f8169e1b3b27bd29/B is initiating minor compaction (all files) 2023-04-09 09:53:55,103 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/75142d4b35234b77b8b69143120ae0cf, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/2457b724b07a4d4baddda6fb98698332, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/2f7e0c607cc443908347c3ba6b5420bf, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/bd5328ad650a4d6cace5df36c0be431c] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp, totalSize=52.9 K 2023-04-09 09:53:55,103 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1732612c92cd83e5f8169e1b3b27bd29/B in TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:55,103 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/bc215c92b27f4bfcb2182f9fee5eeaa0, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/fd85c1ef748648379e8c0bbb223a2862, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/c000684a5bd5435da4125fafd3314f0d, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/793f7129fbc84e67a8d6eb9b658b2db3] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp, totalSize=48.1 K 2023-04-09 09:53:55,104 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 75142d4b35234b77b8b69143120ae0cf, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1681034033276 2023-04-09 09:53:55,104 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting bc215c92b27f4bfcb2182f9fee5eeaa0, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1681034033276 2023-04-09 09:53:55,104 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,111 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting fd85c1ef748648379e8c0bbb223a2862, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=466, earliestPutTs=1681034033879 2023-04-09 09:53:55,111 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 2457b724b07a4d4baddda6fb98698332, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=466, earliestPutTs=1681034033879 2023-04-09 09:53:55,112 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting c000684a5bd5435da4125fafd3314f0d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=480, earliestPutTs=1681034033966 2023-04-09 09:53:55,112 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 2f7e0c607cc443908347c3ba6b5420bf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=480, earliestPutTs=1681034033966 2023-04-09 09:53:55,112 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 793f7129fbc84e67a8d6eb9b658b2db3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=502, earliestPutTs=1681034034333 2023-04-09 09:53:55,112 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting bd5328ad650a4d6cace5df36c0be431c, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=502, earliestPutTs=1681034034333 2023-04-09 09:53:55,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 1732612c92cd83e5f8169e1b3b27bd29 2023-04-09 09:53:55,114 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1732612c92cd83e5f8169e1b3b27bd29 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-09 09:53:55,114 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:55,114 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:55,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:55,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:55,115 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,115 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:55,115 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:55,115 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,115 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:55,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:53:55,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:53:55,115 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:55,115 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:55,115 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:55,115 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:53:55,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:55,115 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:53:55,116 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=A 2023-04-09 09:53:55,116 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:53:55,116 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=B 2023-04-09 09:53:55,116 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:53:55,116 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=C 2023-04-09 09:53:55,116 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:53:55,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:55,139 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): 1732612c92cd83e5f8169e1b3b27bd29#B#compaction#62 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:53:55,141 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1732612c92cd83e5f8169e1b3b27bd29#A#compaction#61 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:53:55,161 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=516 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/A/b2ea89d969a14aeeb324664a81c8a981 2023-04-09 09:53:55,167 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:55,167 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:55,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39920 deadline: 1681034095163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39954 deadline: 1681034095163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,193 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/B/d6a362a1e5084836a6db58515a58a467 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/d6a362a1e5084836a6db58515a58a467 2023-04-09 09:53:55,194 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=516 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/B/cbd529705f4a4f0ab1d62cd9aa3c4c98 2023-04-09 09:53:55,202 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1732612c92cd83e5f8169e1b3b27bd29/B of 1732612c92cd83e5f8169e1b3b27bd29 into d6a362a1e5084836a6db58515a58a467(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:53:55,202 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1732612c92cd83e5f8169e1b3b27bd29: 2023-04-09 09:53:55,202 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29., storeName=1732612c92cd83e5f8169e1b3b27bd29/B, priority=12, startTime=1681034035101; duration=0sec 2023-04-09 09:53:55,202 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:53:55,203 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-09 09:53:55,205 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49290 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-09 09:53:55,205 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): 1732612c92cd83e5f8169e1b3b27bd29/C is initiating minor compaction (all files) 2023-04-09 09:53:55,205 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1732612c92cd83e5f8169e1b3b27bd29/C in TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:55,205 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/2b88ce5648544df1accedbf91c43d3b5, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/06ac5987d7274e5db28161b21b47152a, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/53c65dcf65d041caa45cc6c4b0aecf52, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/6aeaeb44f4c448148933064a9471b2cd] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp, totalSize=48.1 K 2023-04-09 09:53:55,206 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 2b88ce5648544df1accedbf91c43d3b5, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1681034033276 2023-04-09 09:53:55,206 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 06ac5987d7274e5db28161b21b47152a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=466, earliestPutTs=1681034033879 2023-04-09 09:53:55,208 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 53c65dcf65d041caa45cc6c4b0aecf52, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=480, earliestPutTs=1681034033966 2023-04-09 09:53:55,209 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 6aeaeb44f4c448148933064a9471b2cd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=502, earliestPutTs=1681034034333 2023-04-09 09:53:55,215 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:53:55,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:53:55,219 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:53:55,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:53:55,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:53:55,221 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:53:55,221 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:55,221 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): 1732612c92cd83e5f8169e1b3b27bd29#C#compaction#65 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:53:55,221 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:53:55,221 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:53:55,221 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:55,222 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:55,223 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,223 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:55,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:55,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:55,223 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,223 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:53:55,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:55,224 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:55,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:53:55,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:55,224 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:53:55,232 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:53:55,232 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:53:55,232 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:53:55,232 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:55,232 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:53:55,239 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:53:55,239 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:53:55,239 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:55,239 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. started... 2023-04-09 09:53:55,239 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. as already flushing 2023-04-09 09:53:55,239 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:55,240 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:53:55,240 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:53:55,240 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,240 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:53:55,240 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:53:55,242 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:55,242 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:55,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:55,242 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:55,242 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:53:55,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:55,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:55,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:55,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:55,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:53:55,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54379392[Count = 0] remaining members to acquire global barrier 2023-04-09 09:53:55,246 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:53:55,246 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:55,247 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/C/5bb9109228f749d2b046a803fcf8e651 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/5bb9109228f749d2b046a803fcf8e651 2023-04-09 09:53:55,248 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:55,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:55,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:55,248 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:53:55,248 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:53:55,248 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:53:55,248 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,248 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:53:55,250 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:53:55,251 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,251 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:55,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:55,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:55,251 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:53:55,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:55,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:55,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:55,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:55,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:55,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,255 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1732612c92cd83e5f8169e1b3b27bd29/C of 1732612c92cd83e5f8169e1b3b27bd29 into 5bb9109228f749d2b046a803fcf8e651(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:53:55,255 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1732612c92cd83e5f8169e1b3b27bd29: 2023-04-09 09:53:55,255 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29., storeName=1732612c92cd83e5f8169e1b3b27bd29/C, priority=12, startTime=1681034035101; duration=0sec 2023-04-09 09:53:55,255 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:53:55,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:53:55,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:53:55,256 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:53:55,256 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:53:55,256 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:53:55,256 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:53:55,256 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:53:55,258 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:55,258 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:55,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:55,258 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:55,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:55,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:55,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:55,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:55,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:55,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:55,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:55,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:55,259 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:55,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:55,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:55,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,262 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:55,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:55,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,264 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,264 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:55,264 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:55,264 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,265 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:55,265 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:55,265 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:55,265 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:55,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:53:55,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:53:55,265 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:55,265 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:55,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:55,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:55,265 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:53:55,265 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:53:55,265 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:53:55,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:55,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:55,272 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:55,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39920 deadline: 1681034095272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,277 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:55,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39954 deadline: 1681034095276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,365 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:53:55,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:53:55,368 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:53:55,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:53:55,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:53:55,369 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:53:55,369 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:55,369 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:53:55,369 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:53:55,371 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:55,371 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:55,372 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:55,372 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:55,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:55,373 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,373 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:53:55,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:55,373 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:55,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:53:55,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:55,374 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:53:55,374 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:53:55,374 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:53:55,374 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:53:55,374 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:55,374 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:53:55,376 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:53:55,376 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:53:55,376 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:55,376 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. started... 2023-04-09 09:53:55,376 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. as already flushing 2023-04-09 09:53:55,376 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:55,376 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:53:55,376 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:53:55,376 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,376 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:53:55,376 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:53:55,380 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,380 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:55,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:55,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:55,381 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:55,381 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:53:55,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:55,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:55,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:55,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:55,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:53:55,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47c37d48[Count = 0] remaining members to acquire global barrier 2023-04-09 09:53:55,383 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:53:55,383 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:55,384 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:55,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:55,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:55,384 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:53:55,384 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:53:55,384 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:53:55,384 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,384 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:53:55,386 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,386 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:53:55,386 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:55,386 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:53:55,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:55,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:55,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:55,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:55,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:55,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:55,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:55,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:53:55,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:53:55,390 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:53:55,390 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:53:55,390 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:53:55,390 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:53:55,390 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:53:55,393 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:55,393 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:55,393 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:55,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:55,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:55,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:55,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:55,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:55,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:55,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:55,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:55,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:55,394 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:55,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:55,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:55,397 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:55,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:55,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,404 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,404 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:55,404 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:55,404 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,404 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:55,404 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:55,404 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:55,404 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:55,404 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:55,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:53:55,405 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:53:55,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:55,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:55,404 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:55,405 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:53:55,405 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:53:55,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:55,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:55,405 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:53:55,480 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:55,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39920 deadline: 1681034095477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,480 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:55,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39954 deadline: 1681034095478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,500 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:55,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39864 deadline: 1681034095498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,505 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:53:55,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:53:55,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:53:55,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:53:55,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:53:55,509 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:53:55,509 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:55,509 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:53:55,509 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:53:55,509 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:55,509 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:55,511 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,511 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:55,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:55,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:55,511 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,511 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:53:55,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:55,511 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:55,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:53:55,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:55,512 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:53:55,512 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:53:55,512 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:53:55,512 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:53:55,512 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:55,512 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:53:55,512 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:53:55,513 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:53:55,513 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:55,513 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. started... 2023-04-09 09:53:55,513 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. as already flushing 2023-04-09 09:53:55,513 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:55,513 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:53:55,513 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:53:55,513 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,513 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:53:55,513 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:53:55,516 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,516 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:55,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:55,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:55,516 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:55,516 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:53:55,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:55,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:55,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:55,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:55,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:53:55,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b9ca5ec[Count = 0] remaining members to acquire global barrier 2023-04-09 09:53:55,518 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:53:55,518 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:55,519 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:55,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:55,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:55,519 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:53:55,520 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:53:55,520 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:53:55,520 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,520 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:53:55,522 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:53:55,522 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,522 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:55,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:55,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:55,522 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:53:55,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:55,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:55,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:55,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:55,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:55,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:53:55,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:53:55,525 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:53:55,525 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:53:55,525 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:53:55,525 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:53:55,525 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:53:55,527 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:55,527 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:55,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:55,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:55,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:55,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:55,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:55,527 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:55,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:55,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:55,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:55,528 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:55,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:55,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:55,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:55,529 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:55,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:55,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,532 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:55,532 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:55,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:55,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:55,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:55,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:55,536 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,536 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:55,536 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:55,536 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,536 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:55,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:53:55,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:53:55,536 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:55,536 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:55,536 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:55,537 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:53:55,537 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:53:55,537 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:53:55,601 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/A/27e423631ae44e398a0385f92689488e as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/27e423631ae44e398a0385f92689488e 2023-04-09 09:53:55,610 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1732612c92cd83e5f8169e1b3b27bd29/A of 1732612c92cd83e5f8169e1b3b27bd29 into 27e423631ae44e398a0385f92689488e(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:53:55,610 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1732612c92cd83e5f8169e1b3b27bd29: 2023-04-09 09:53:55,610 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29., storeName=1732612c92cd83e5f8169e1b3b27bd29/A, priority=12, startTime=1681034035101; duration=0sec 2023-04-09 09:53:55,610 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:53:55,626 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=516 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/C/0e9b5ba1cfdc411096d1337f79fb345a 2023-04-09 09:53:55,632 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/A/b2ea89d969a14aeeb324664a81c8a981 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/b2ea89d969a14aeeb324664a81c8a981 2023-04-09 09:53:55,637 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:53:55,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:53:55,639 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/b2ea89d969a14aeeb324664a81c8a981, entries=250, sequenceid=516, filesize=16.7 K 2023-04-09 09:53:55,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:53:55,640 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/B/cbd529705f4a4f0ab1d62cd9aa3c4c98 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/cbd529705f4a4f0ab1d62cd9aa3c4c98 2023-04-09 09:53:55,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:53:55,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:53:55,644 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:53:55,644 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:55,644 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:53:55,644 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:53:55,646 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/cbd529705f4a4f0ab1d62cd9aa3c4c98, entries=150, sequenceid=516, filesize=11.9 K 2023-04-09 09:53:55,647 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/C/0e9b5ba1cfdc411096d1337f79fb345a as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/0e9b5ba1cfdc411096d1337f79fb345a 2023-04-09 09:53:55,648 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:55,648 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:55,649 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:55,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:55,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:55,649 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,650 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,650 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:53:55,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:55,650 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:55,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:53:55,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:55,651 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:53:55,651 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:53:55,651 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:53:55,651 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:53:55,651 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:55,651 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:53:55,651 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:53:55,651 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:53:55,651 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:55,652 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. started... 2023-04-09 09:53:55,652 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. as already flushing 2023-04-09 09:53:55,652 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:55,652 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:53:55,652 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:53:55,652 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,652 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:53:55,652 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:53:55,653 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,653 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:55,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:55,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:55,654 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/0e9b5ba1cfdc411096d1337f79fb345a, entries=150, sequenceid=516, filesize=11.9 K 2023-04-09 09:53:55,654 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:55,654 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:53:55,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:55,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:55,655 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 1732612c92cd83e5f8169e1b3b27bd29 in 541ms, sequenceid=516, compaction requested=false 2023-04-09 09:53:55,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:55,655 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1732612c92cd83e5f8169e1b3b27bd29: 2023-04-09 09:53:55,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:55,656 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:53:55,656 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18204211[Count = 0] remaining members to acquire global barrier 2023-04-09 09:53:55,656 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:53:55,657 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:55,658 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:55,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:55,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:55,658 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,658 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:53:55,659 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:53:55,659 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:53:55,659 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:53:55,661 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,661 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:53:55,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:55,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:55,661 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:55,661 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:53:55,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:55,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:55,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:55,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:55,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:55,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:53:55,663 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:53:55,663 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:53:55,663 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:53:55,663 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:53:55,663 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:53:55,663 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:53:55,665 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:55,665 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:55,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:55,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:55,665 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:55,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:55,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:55,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:55,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:55,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:55,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:55,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:55,666 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:55,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:55,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:55,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,667 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:55,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:55,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,670 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,670 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:55,670 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:55,670 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,670 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:55,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:53:55,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:53:55,670 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:55,670 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:55,670 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:55,670 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:53:55,670 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:53:55,670 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:55,670 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:55,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:55,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:55,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:55,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:55,770 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:53:55,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:53:55,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:53:55,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:53:55,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:53:55,784 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:53:55,784 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:55,784 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:53:55,784 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:53:55,785 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:55,785 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:55,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 1732612c92cd83e5f8169e1b3b27bd29 2023-04-09 09:53:55,786 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1732612c92cd83e5f8169e1b3b27bd29 3/3 column families, dataSize=114.05 KB heapSize=339.41 KB 2023-04-09 09:53:55,786 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:55,786 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:55,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:55,786 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681034035147, max timestamp=1681034035784Num uniques 859; 2023-04-09 09:53:55,787 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-09 09:53:55,787 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,787 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:53:55,787 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=A 2023-04-09 09:53:55,787 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:53:55,787 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=B 2023-04-09 09:53:55,787 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:53:55,787 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=C 2023-04-09 09:53:55,787 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:53:55,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:55,791 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:55,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:53:55,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:55,792 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:53:55,792 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:53:55,792 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:53:55,792 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:53:55,792 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:55,792 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:53:55,796 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:53:55,796 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:53:55,796 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:55,796 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. started... 2023-04-09 09:53:55,797 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. as already flushing 2023-04-09 09:53:55,797 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:55,797 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:53:55,797 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:53:55,797 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,797 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:53:55,797 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:53:55,801 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,801 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:55,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:55,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:55,801 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:55,801 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:53:55,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:55,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:55,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:55,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:55,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:53:55,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7063e984[Count = 0] remaining members to acquire global barrier 2023-04-09 09:53:55,806 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:53:55,806 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:55,808 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:55,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:55,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:55,808 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:53:55,808 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:53:55,808 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:53:55,809 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,809 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:53:55,810 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:53:55,810 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:55,810 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:53:55,811 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:55,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:55,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:55,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:55,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:55,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:55,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:55,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:53:55,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:53:55,815 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:53:55,815 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:53:55,815 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:53:55,815 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:53:55,815 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:53:55,818 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:55,818 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:55,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:55,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:55,818 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:55,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:55,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:55,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:55,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:55,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:55,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:55,819 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:55,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:55,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:55,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:55,821 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:55,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:55,823 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:55,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39954 deadline: 1681034095821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,826 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:55,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39920 deadline: 1681034095824, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,830 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,830 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:55,830 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:55,830 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,830 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:55,830 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:53:55,830 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:53:55,830 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:53:55,830 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:55,831 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:55,830 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:55,831 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:53:55,831 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:53:55,831 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:55,831 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:55,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:55,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:55,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:55,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:55,926 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:55,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39954 deadline: 1681034095926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,930 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:55,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39920 deadline: 1681034095927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,931 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:53:55,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:53:55,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:53:55,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:53:55,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:53:55,941 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:53:55,941 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:55,941 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:53:55,941 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:53:55,943 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:55,943 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:55,945 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:55,945 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,946 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,946 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:53:55,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:55,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:55,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:55,948 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:55,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:53:55,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:55,948 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:53:55,948 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:53:55,948 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:53:55,948 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:53:55,949 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:55,949 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:53:55,949 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:53:55,949 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:53:55,949 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:55,949 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. started... 2023-04-09 09:53:55,949 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. as already flushing 2023-04-09 09:53:55,949 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:55,949 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:53:55,949 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:53:55,949 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,949 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:53:55,949 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:53:55,951 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:55,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:55,952 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:55,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:55,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:55,953 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:55,953 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:53:55,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:55,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:55,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:53:55,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c16e3fd[Count = 0] remaining members to acquire global barrier 2023-04-09 09:53:55,954 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:53:55,955 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:55,957 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:55,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:55,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:55,957 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:53:55,957 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:53:55,957 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:53:55,958 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,958 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:53:55,959 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:53:55,959 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,959 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:55,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:55,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:55,959 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:53:55,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:55,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:55,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:55,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:55,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:55,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:53:55,963 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:53:55,963 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:53:55,963 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:53:55,963 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:53:55,963 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:53:55,963 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:53:55,965 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:55,965 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:55,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:55,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:55,965 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:55,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:55,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:55,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:55,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:55,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:55,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:55,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:55,966 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:55,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:55,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:55,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,968 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:55,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:55,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,972 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:55,972 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:55,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:55,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:55,972 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,972 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:55,972 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:55,972 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:55,972 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:55,972 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:55,972 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:55,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:55,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:55,973 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:55,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:53:55,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:53:55,973 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:53:55,973 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:53:56,073 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:53:56,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:53:56,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:53:56,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:53:56,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:53:56,078 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:53:56,078 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:56,078 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:53:56,078 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:53:56,078 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:56,078 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:56,079 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:56,079 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:56,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:56,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:56,080 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:56,080 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:53:56,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:56,080 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:56,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:53:56,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:56,081 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:53:56,081 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:53:56,081 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:53:56,081 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:56,081 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:53:56,081 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:53:56,081 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:53:56,081 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:56,081 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. started... 2023-04-09 09:53:56,081 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. as already flushing 2023-04-09 09:53:56,081 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:56,082 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:53:56,082 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:53:56,082 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:56,082 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:53:56,082 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:53:56,083 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:56,083 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:56,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:56,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:56,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:56,083 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:56,084 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:53:56,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:56,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:56,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:56,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:56,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:56,085 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:53:56,085 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@121b8056[Count = 0] remaining members to acquire global barrier 2023-04-09 09:53:56,085 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:53:56,085 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:56,086 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:56,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:56,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:56,086 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:53:56,086 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:53:56,086 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:53:56,086 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:56,086 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:53:56,088 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:56,088 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:53:56,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:56,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:56,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:56,089 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:56,089 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:53:56,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:56,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:56,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:56,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:56,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:56,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:56,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:56,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:53:56,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:53:56,091 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:53:56,091 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:53:56,091 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:53:56,091 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:53:56,091 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:53:56,093 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:56,093 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:56,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:56,093 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:56,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:56,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:56,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:56,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:56,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:56,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:56,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:56,093 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:56,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:56,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:56,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:56,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:56,095 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:56,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:56,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:56,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:56,097 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:56,097 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:56,097 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:56,097 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:56,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:53:56,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:53:56,097 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:56,097 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:53:56,097 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:56,098 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:56,098 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:56,098 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:53:56,098 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:53:56,100 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:56,100 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:56,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:56,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:56,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:56,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:56,129 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:56,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39954 deadline: 1681034096129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:56,134 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:56,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39920 deadline: 1681034096134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:56,198 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:53:56,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:53:56,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:53:56,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:53:56,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:53:56,203 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:53:56,203 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:56,203 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:53:56,203 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:53:56,204 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:56,204 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:56,205 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:56,205 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:56,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:56,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:56,205 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:56,205 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:53:56,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:56,206 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:56,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:53:56,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:56,206 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:53:56,207 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:53:56,207 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:53:56,207 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:53:56,207 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:56,207 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:53:56,207 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:53:56,207 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:53:56,207 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:56,208 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. started... 2023-04-09 09:53:56,208 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. as already flushing 2023-04-09 09:53:56,208 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:56,208 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:53:56,208 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:53:56,208 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:56,208 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:53:56,208 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:53:56,209 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=540 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/A/afadaee9accd445f81a0be9b4d8340a2 2023-04-09 09:53:56,211 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:56,211 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:56,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:56,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:56,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:56,211 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:56,211 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:53:56,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:56,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:56,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:56,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:56,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:56,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:53:56,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@83178ce[Count = 0] remaining members to acquire global barrier 2023-04-09 09:53:56,215 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:53:56,215 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:56,216 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:56,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:56,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:56,216 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:53:56,216 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:53:56,216 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:53:56,216 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:56,216 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:53:56,218 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:53:56,218 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:56,218 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:56,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:56,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:56,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:56,218 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:53:56,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:56,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:56,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:56,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:56,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:56,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:56,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:56,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:53:56,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:53:56,222 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:53:56,222 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:53:56,222 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:53:56,222 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:53:56,222 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:53:56,223 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:56,224 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:56,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:56,224 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:56,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:56,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:56,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:56,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:56,224 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:56,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:56,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:56,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:56,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:56,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:56,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:56,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:56,226 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:56,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:56,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:56,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:56,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:56,228 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=540 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/B/487f0ee29be8415c8775d4e605574f48 2023-04-09 09:53:56,230 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:56,230 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:56,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:56,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:56,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:56,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:56,230 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:56,231 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:56,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:53:56,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:53:56,231 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:56,231 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:53:56,231 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:53:56,231 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:56,232 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:56,232 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:56,232 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:56,232 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:56,232 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:53:56,249 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=540 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/C/33407d10fa614cef843b4cab61bbddb0 2023-04-09 09:53:56,257 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/A/afadaee9accd445f81a0be9b4d8340a2 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/afadaee9accd445f81a0be9b4d8340a2 2023-04-09 09:53:56,263 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/afadaee9accd445f81a0be9b4d8340a2, entries=200, sequenceid=540, filesize=14.3 K 2023-04-09 09:53:56,264 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/B/487f0ee29be8415c8775d4e605574f48 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/487f0ee29be8415c8775d4e605574f48 2023-04-09 09:53:56,271 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/487f0ee29be8415c8775d4e605574f48, entries=150, sequenceid=540, filesize=11.9 K 2023-04-09 09:53:56,272 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/C/33407d10fa614cef843b4cab61bbddb0 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/33407d10fa614cef843b4cab61bbddb0 2023-04-09 09:53:56,277 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/33407d10fa614cef843b4cab61bbddb0, entries=150, sequenceid=540, filesize=11.9 K 2023-04-09 09:53:56,278 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~91.69 KB/93890, heapSize ~267.14 KB/273552, currentSize=87.22 KB/89310 for 1732612c92cd83e5f8169e1b3b27bd29 in 493ms, sequenceid=540, compaction requested=true 2023-04-09 09:53:56,278 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1732612c92cd83e5f8169e1b3b27bd29: 2023-04-09 09:53:56,279 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:53:56,279 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-09 09:53:56,279 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-09 09:53:56,279 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:53:56,279 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:53:56,280 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 44560 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-09 09:53:56,280 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1912): 1732612c92cd83e5f8169e1b3b27bd29/A is initiating minor compaction (all files) 2023-04-09 09:53:56,280 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1732612c92cd83e5f8169e1b3b27bd29/A in TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:56,280 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/27e423631ae44e398a0385f92689488e, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/b2ea89d969a14aeeb324664a81c8a981, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/afadaee9accd445f81a0be9b4d8340a2] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp, totalSize=43.5 K 2023-04-09 09:53:56,281 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37240 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-09 09:53:56,281 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): 1732612c92cd83e5f8169e1b3b27bd29/B is initiating minor compaction (all files) 2023-04-09 09:53:56,281 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1732612c92cd83e5f8169e1b3b27bd29/B in TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:56,281 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/d6a362a1e5084836a6db58515a58a467, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/cbd529705f4a4f0ab1d62cd9aa3c4c98, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/487f0ee29be8415c8775d4e605574f48] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp, totalSize=36.4 K 2023-04-09 09:53:56,281 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 27e423631ae44e398a0385f92689488e, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=502, earliestPutTs=1681034034333 2023-04-09 09:53:56,281 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting d6a362a1e5084836a6db58515a58a467, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=502, earliestPutTs=1681034034333 2023-04-09 09:53:56,282 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting b2ea89d969a14aeeb324664a81c8a981, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=516, earliestPutTs=1681034034466 2023-04-09 09:53:56,282 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting cbd529705f4a4f0ab1d62cd9aa3c4c98, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=516, earliestPutTs=1681034034466 2023-04-09 09:53:56,282 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting afadaee9accd445f81a0be9b4d8340a2, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=540, earliestPutTs=1681034035161 2023-04-09 09:53:56,283 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 487f0ee29be8415c8775d4e605574f48, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=540, earliestPutTs=1681034035161 2023-04-09 09:53:56,291 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1732612c92cd83e5f8169e1b3b27bd29#A#compaction#69 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:53:56,291 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): 1732612c92cd83e5f8169e1b3b27bd29#B#compaction#70 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:53:56,331 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:53:56,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:53:56,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:53:56,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:53:56,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:53:56,337 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:53:56,337 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:56,337 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:53:56,337 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:53:56,338 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:56,338 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:56,342 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:56,342 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:56,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:56,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:56,342 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:56,342 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:53:56,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:56,342 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:56,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:53:56,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:56,343 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:53:56,343 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:53:56,343 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:53:56,343 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:53:56,343 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:56,343 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:53:56,343 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:53:56,343 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:53:56,344 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:56,344 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. started... 2023-04-09 09:53:56,344 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1732612c92cd83e5f8169e1b3b27bd29 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-09 09:53:56,344 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=A 2023-04-09 09:53:56,344 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:53:56,344 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=B 2023-04-09 09:53:56,344 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:53:56,344 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=C 2023-04-09 09:53:56,344 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:53:56,354 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=556 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/A/d8a88c37a5164d61a7ccae323fe3ed2d 2023-04-09 09:53:56,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 1732612c92cd83e5f8169e1b3b27bd29 2023-04-09 09:53:56,435 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. as already flushing 2023-04-09 09:53:56,459 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:56,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39920 deadline: 1681034096457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:56,459 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:56,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39954 deadline: 1681034096459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:56,509 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:56,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39864 deadline: 1681034096507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:56,561 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:56,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39954 deadline: 1681034096560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:56,566 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:56,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39920 deadline: 1681034096563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:56,705 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/B/327f26834c044e32a9d079e451bb7db1 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/327f26834c044e32a9d079e451bb7db1 2023-04-09 09:53:56,713 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1732612c92cd83e5f8169e1b3b27bd29/B of 1732612c92cd83e5f8169e1b3b27bd29 into 327f26834c044e32a9d079e451bb7db1(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:53:56,714 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1732612c92cd83e5f8169e1b3b27bd29: 2023-04-09 09:53:56,714 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29., storeName=1732612c92cd83e5f8169e1b3b27bd29/B, priority=13, startTime=1681034036279; duration=0sec 2023-04-09 09:53:56,714 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:53:56,714 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-09 09:53:56,715 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37240 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-09 09:53:56,716 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): 1732612c92cd83e5f8169e1b3b27bd29/C is initiating minor compaction (all files) 2023-04-09 09:53:56,716 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1732612c92cd83e5f8169e1b3b27bd29/C in TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:56,716 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/5bb9109228f749d2b046a803fcf8e651, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/0e9b5ba1cfdc411096d1337f79fb345a, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/33407d10fa614cef843b4cab61bbddb0] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp, totalSize=36.4 K 2023-04-09 09:53:56,716 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 5bb9109228f749d2b046a803fcf8e651, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=502, earliestPutTs=1681034034333 2023-04-09 09:53:56,717 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 0e9b5ba1cfdc411096d1337f79fb345a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=516, earliestPutTs=1681034034466 2023-04-09 09:53:56,718 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 33407d10fa614cef843b4cab61bbddb0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=540, earliestPutTs=1681034035161 2023-04-09 09:53:56,718 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/A/a602cab7afbe48cd9969f40505daad0b as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/a602cab7afbe48cd9969f40505daad0b 2023-04-09 09:53:56,726 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1732612c92cd83e5f8169e1b3b27bd29/A of 1732612c92cd83e5f8169e1b3b27bd29 into a602cab7afbe48cd9969f40505daad0b(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:53:56,726 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1732612c92cd83e5f8169e1b3b27bd29: 2023-04-09 09:53:56,726 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29., storeName=1732612c92cd83e5f8169e1b3b27bd29/A, priority=13, startTime=1681034036279; duration=0sec 2023-04-09 09:53:56,726 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:53:56,727 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): 1732612c92cd83e5f8169e1b3b27bd29#C#compaction#73 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:53:56,743 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/C/c282ada0bf7948f0a1813c115747f468 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/c282ada0bf7948f0a1813c115747f468 2023-04-09 09:53:56,749 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1732612c92cd83e5f8169e1b3b27bd29/C of 1732612c92cd83e5f8169e1b3b27bd29 into c282ada0bf7948f0a1813c115747f468(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:53:56,749 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1732612c92cd83e5f8169e1b3b27bd29: 2023-04-09 09:53:56,749 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29., storeName=1732612c92cd83e5f8169e1b3b27bd29/C, priority=13, startTime=1681034036279; duration=0sec 2023-04-09 09:53:56,750 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:53:56,764 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:56,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39954 deadline: 1681034096763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:56,769 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:56,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39920 deadline: 1681034096768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:56,770 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=556 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/B/d5ce30c0e28b40f4aa1ad2b26d20cc2b 2023-04-09 09:53:57,069 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:57,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39954 deadline: 1681034097067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:57,070 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:57,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39920 deadline: 1681034097070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:57,196 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=556 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/C/a17924d94fa9457e89574fc641c9ffb4 2023-04-09 09:53:57,204 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/A/d8a88c37a5164d61a7ccae323fe3ed2d as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/d8a88c37a5164d61a7ccae323fe3ed2d 2023-04-09 09:53:57,209 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/d8a88c37a5164d61a7ccae323fe3ed2d, entries=150, sequenceid=556, filesize=11.9 K 2023-04-09 09:53:57,210 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/B/d5ce30c0e28b40f4aa1ad2b26d20cc2b as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/d5ce30c0e28b40f4aa1ad2b26d20cc2b 2023-04-09 09:53:57,216 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/d5ce30c0e28b40f4aa1ad2b26d20cc2b, entries=150, sequenceid=556, filesize=11.9 K 2023-04-09 09:53:57,218 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/C/a17924d94fa9457e89574fc641c9ffb4 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/a17924d94fa9457e89574fc641c9ffb4 2023-04-09 09:53:57,225 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/a17924d94fa9457e89574fc641c9ffb4, entries=150, sequenceid=556, filesize=11.9 K 2023-04-09 09:53:57,226 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 1732612c92cd83e5f8169e1b3b27bd29 in 882ms, sequenceid=556, compaction requested=false 2023-04-09 09:53:57,226 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1732612c92cd83e5f8169e1b3b27bd29: 2023-04-09 09:53:57,226 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:57,226 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:53:57,226 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:53:57,227 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:57,227 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:53:57,227 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:53:57,228 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:57,228 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:57,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:57,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:57,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:57,229 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:57,229 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:53:57,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:57,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:57,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:57,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:57,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:57,230 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:53:57,230 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67f19f85[Count = 0] remaining members to acquire global barrier 2023-04-09 09:53:57,230 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:53:57,231 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:57,231 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:57,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:57,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:57,232 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:53:57,232 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:53:57,232 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:53:57,232 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:57,232 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:53:57,235 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:57,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:57,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:57,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:57,235 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:53:57,235 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:57,235 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:53:57,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:57,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:57,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:57,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:57,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:57,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:57,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:57,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:53:57,238 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:53:57,238 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:53:57,238 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:53:57,238 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:53:57,238 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:53:57,238 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:53:57,240 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:57,240 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:57,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:57,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:57,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:57,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:57,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:57,240 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:57,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:57,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:57,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:57,241 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:57,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:57,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:57,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:57,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:57,242 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:57,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:57,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:57,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:57,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:57,245 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:57,245 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:57,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:57,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:57,245 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:57,245 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:57,245 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:57,245 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:57,245 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:57,245 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:57,245 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:57,245 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:53:57,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:57,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:57,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:53:57,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:53:57,245 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:57,245 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:53:57,245 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:53:57,346 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:53:57,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:53:57,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:53:57,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:53:57,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:53:57,349 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:53:57,350 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:57,350 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:53:57,350 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:53:57,350 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:57,350 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:57,353 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:57,353 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:57,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:57,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:57,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:57,354 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:57,354 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:53:57,354 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:57,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:53:57,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:57,355 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:53:57,360 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:53:57,360 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:53:57,360 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:53:57,360 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:57,360 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:53:57,372 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:53:57,372 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:53:57,372 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:57,372 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. started... 2023-04-09 09:53:57,372 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1732612c92cd83e5f8169e1b3b27bd29 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-09 09:53:57,373 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=A 2023-04-09 09:53:57,373 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:53:57,373 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=B 2023-04-09 09:53:57,373 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:53:57,373 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=C 2023-04-09 09:53:57,373 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:53:57,400 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=575 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/A/95c0fd3565da4dacbbf1be2d1b695e35 2023-04-09 09:53:57,421 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=575 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/B/3582356e9b9a47eebefb2628826dc5a8 2023-04-09 09:53:57,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 1732612c92cd83e5f8169e1b3b27bd29 2023-04-09 09:53:57,573 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. as already flushing 2023-04-09 09:53:57,594 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:57,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39954 deadline: 1681034097590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:57,594 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:57,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39920 deadline: 1681034097590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:57,696 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:57,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39954 deadline: 1681034097695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:57,697 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:57,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39920 deadline: 1681034097695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:57,838 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=575 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/C/c4a1ea2c4d614a5bad9db3d9bf95b240 2023-04-09 09:53:57,845 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/A/95c0fd3565da4dacbbf1be2d1b695e35 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/95c0fd3565da4dacbbf1be2d1b695e35 2023-04-09 09:53:57,851 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/95c0fd3565da4dacbbf1be2d1b695e35, entries=150, sequenceid=575, filesize=11.9 K 2023-04-09 09:53:57,852 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/B/3582356e9b9a47eebefb2628826dc5a8 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/3582356e9b9a47eebefb2628826dc5a8 2023-04-09 09:53:57,859 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/3582356e9b9a47eebefb2628826dc5a8, entries=150, sequenceid=575, filesize=11.9 K 2023-04-09 09:53:57,860 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/C/c4a1ea2c4d614a5bad9db3d9bf95b240 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/c4a1ea2c4d614a5bad9db3d9bf95b240 2023-04-09 09:53:57,865 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/c4a1ea2c4d614a5bad9db3d9bf95b240, entries=150, sequenceid=575, filesize=11.9 K 2023-04-09 09:53:57,866 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=93.93 KB/96180 for 1732612c92cd83e5f8169e1b3b27bd29 in 494ms, sequenceid=575, compaction requested=true 2023-04-09 09:53:57,866 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1732612c92cd83e5f8169e1b3b27bd29: 2023-04-09 09:53:57,866 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:57,866 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:53:57,866 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:53:57,866 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:57,866 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:53:57,866 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:53:57,868 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:57,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:57,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:57,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:57,868 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:57,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:57,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:57,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:57,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:57,870 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:57,870 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:53:57,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:57,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:53:57,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ce3818d[Count = 0] remaining members to acquire global barrier 2023-04-09 09:53:57,870 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:53:57,870 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:57,876 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:57,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:57,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:57,876 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:53:57,876 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:53:57,876 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:53:57,876 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:57,876 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:53:57,877 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:57,877 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:53:57,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:57,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:57,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:57,877 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:57,877 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:53:57,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:57,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:57,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:57,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:57,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:57,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:57,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:57,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:53:57,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:53:57,880 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:53:57,880 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:53:57,880 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:53:57,880 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:53:57,880 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:53:57,881 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:57,881 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:57,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:57,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:57,882 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:57,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:57,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:57,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:57,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:57,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:57,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:57,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:57,883 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:57,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:57,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:57,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:57,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:57,884 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:57,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:57,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:57,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:57,886 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:57,886 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:57,886 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:57,886 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:57,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:57,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:57,886 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:53:57,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:53:57,886 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:57,887 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:57,887 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:57,887 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:57,887 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:57,887 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:57,887 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:53:57,887 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:53:57,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:57,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:57,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 1732612c92cd83e5f8169e1b3b27bd29 2023-04-09 09:53:57,899 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1732612c92cd83e5f8169e1b3b27bd29 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-09 09:53:57,901 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=A 2023-04-09 09:53:57,901 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:53:57,901 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=B 2023-04-09 09:53:57,901 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:53:57,901 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=C 2023-04-09 09:53:57,901 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:53:57,947 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:57,947 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:57,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39954 deadline: 1681034097944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:57,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 289 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39920 deadline: 1681034097944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:57,987 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:53:57,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:53:57,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:53:57,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:53:57,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:53:57,995 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:53:57,995 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:57,995 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:53:57,995 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:53:57,996 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:57,996 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:57,998 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:57,998 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:57,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:57,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:57,998 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,000 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:53:58,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:58,000 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:58,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:53:58,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:58,001 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:53:58,001 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:53:58,001 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:53:58,001 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:53:58,001 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:58,001 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:53:58,001 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:53:58,001 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:53:58,001 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:58,001 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. started... 2023-04-09 09:53:58,002 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. as already flushing 2023-04-09 09:53:58,002 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:58,002 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:53:58,002 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:53:58,002 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,002 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:53:58,002 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:53:58,003 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,003 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:58,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:58,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:58,004 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:58,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:58,004 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:53:58,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:58,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:58,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:58,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:53:58,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3234b777[Count = 0] remaining members to acquire global barrier 2023-04-09 09:53:58,006 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:53:58,006 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:58,007 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:58,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:58,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:58,007 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:53:58,007 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:53:58,007 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:53:58,007 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,007 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:53:58,008 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,008 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:53:58,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:58,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:58,009 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:58,009 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:53:58,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:58,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:58,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:58,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:58,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:58,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:53:58,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:53:58,011 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:53:58,011 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:53:58,011 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:53:58,011 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:53:58,011 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:53:58,014 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:58,014 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:58,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:58,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:58,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:58,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:58,014 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:58,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:58,015 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:58,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:58,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:58,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:58,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:58,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:58,016 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:58,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:58,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:58,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,018 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,018 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:58,018 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:58,018 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,018 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:58,018 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:58,018 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:58,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:53:58,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:53:58,019 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:58,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:58,018 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:58,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:58,019 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:58,019 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:53:58,019 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:53:58,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:58,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:58,022 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:53:58,052 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:58,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39954 deadline: 1681034098049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,053 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:58,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 291 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39920 deadline: 1681034098049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,119 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:53:58,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:53:58,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:53:58,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:53:58,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:53:58,122 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:53:58,122 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:58,123 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:53:58,123 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:53:58,123 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:58,123 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:58,124 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:58,124 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:58,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:58,125 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,125 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:53:58,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:58,125 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:58,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:53:58,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:58,126 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:53:58,126 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:53:58,126 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:53:58,126 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:53:58,126 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:58,126 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:53:58,126 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:53:58,126 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:53:58,126 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:58,126 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. started... 2023-04-09 09:53:58,126 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. as already flushing 2023-04-09 09:53:58,127 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:58,127 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:53:58,127 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:53:58,127 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,127 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:53:58,127 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:53:58,128 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,128 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:58,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:58,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:58,128 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:58,128 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:53:58,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:58,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:58,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:58,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:58,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:53:58,130 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:53:58,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31404ef7[Count = 0] remaining members to acquire global barrier 2023-04-09 09:53:58,130 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:58,137 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:58,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:58,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:58,137 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:53:58,137 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:53:58,137 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:53:58,138 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,138 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:53:58,139 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,139 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:53:58,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:58,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:58,139 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:58,139 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:53:58,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:58,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:58,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:58,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:58,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:58,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:53:58,142 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:53:58,142 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:53:58,142 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:53:58,142 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:53:58,142 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:53:58,142 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:53:58,143 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:58,143 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:58,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:58,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:58,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:58,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:58,144 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,144 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:58,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:58,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:58,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:58,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:58,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:58,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:58,145 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:58,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:58,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:58,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:58,148 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,148 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:58,148 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:58,148 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:58,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:58,149 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:53:58,149 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:53:58,149 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:58,149 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,149 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:58,149 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:58,149 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:58,149 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:53:58,149 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:53:58,149 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:58,149 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:53:58,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:58,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:58,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:58,249 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:53:58,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:53:58,252 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:53:58,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:53:58,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:53:58,253 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:53:58,253 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:58,253 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:53:58,253 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:53:58,253 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:58,254 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:58,255 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:58,255 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:58,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:58,255 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,255 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:53:58,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:58,255 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:58,256 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:58,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39954 deadline: 1681034098255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,256 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:58,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 293 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39920 deadline: 1681034098255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:53:58,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:58,256 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:53:58,257 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:53:58,257 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:53:58,257 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:53:58,257 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:58,257 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:53:58,260 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:53:58,260 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:53:58,260 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:58,260 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. started... 2023-04-09 09:53:58,260 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. as already flushing 2023-04-09 09:53:58,260 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:58,260 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:53:58,260 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:53:58,260 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,260 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:53:58,260 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:53:58,262 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,262 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:58,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:58,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:58,262 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:58,262 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:53:58,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:58,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:58,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:58,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:58,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:53:58,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a26e2ed[Count = 0] remaining members to acquire global barrier 2023-04-09 09:53:58,264 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:53:58,264 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:58,265 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:58,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:58,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:58,265 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:53:58,265 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:53:58,265 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:53:58,265 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,265 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:53:58,276 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:53:58,276 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:58,276 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:53:58,278 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:58,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:58,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:58,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:58,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:58,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:58,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:58,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:53:58,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:53:58,281 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:53:58,281 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:53:58,281 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:53:58,281 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:53:58,281 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:53:58,283 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:58,283 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:58,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:58,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:58,283 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:58,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:58,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:58,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:58,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:58,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:58,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:58,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:58,284 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:58,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:58,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:58,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,286 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:58,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:58,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,289 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:58,289 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,289 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:58,289 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:58,289 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,289 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:58,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:53:58,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:53:58,289 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:58,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:58,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:58,289 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:58,290 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:53:58,290 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:53:58,290 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:58,290 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:58,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:58,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:58,310 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=594 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/A/64225f79554c411b9bc55329d68423ab 2023-04-09 09:53:58,349 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=594 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/B/a54c08b9597b429bbc533932d98a7702 2023-04-09 09:53:58,389 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=594 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/C/05dd1935f0174433a4c9bcf3fe0dac69 2023-04-09 09:53:58,390 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:53:58,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:53:58,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:53:58,397 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/A/64225f79554c411b9bc55329d68423ab as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/64225f79554c411b9bc55329d68423ab 2023-04-09 09:53:58,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:53:58,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:53:58,408 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:53:58,408 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:58,408 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:53:58,408 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:53:58,408 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/64225f79554c411b9bc55329d68423ab, entries=200, sequenceid=594, filesize=14.3 K 2023-04-09 09:53:58,409 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:58,409 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:58,410 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/B/a54c08b9597b429bbc533932d98a7702 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/a54c08b9597b429bbc533932d98a7702 2023-04-09 09:53:58,411 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,411 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:58,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:58,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:58,412 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,412 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:53:58,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:58,412 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:58,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:53:58,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:58,413 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:53:58,416 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:53:58,416 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:53:58,416 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:53:58,416 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:58,416 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:53:58,420 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:53:58,420 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:53:58,420 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:58,420 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. started... 2023-04-09 09:53:58,420 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. as already flushing 2023-04-09 09:53:58,420 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:58,420 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:53:58,420 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:53:58,420 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,420 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:53:58,420 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:53:58,423 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,423 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:58,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:58,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:58,424 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:58,424 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:53:58,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:58,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:58,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:58,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:58,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:53:58,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a29ffcd[Count = 0] remaining members to acquire global barrier 2023-04-09 09:53:58,426 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:53:58,426 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:58,427 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:58,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:58,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:58,427 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:53:58,427 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,427 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:53:58,427 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:53:58,427 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:53:58,428 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/a54c08b9597b429bbc533932d98a7702, entries=150, sequenceid=594, filesize=11.9 K 2023-04-09 09:53:58,429 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/C/05dd1935f0174433a4c9bcf3fe0dac69 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/05dd1935f0174433a4c9bcf3fe0dac69 2023-04-09 09:53:58,431 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:53:58,431 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,431 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:58,431 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:53:58,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:58,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:58,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:58,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:58,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:58,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:58,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:58,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:53:58,434 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:53:58,435 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:53:58,435 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:53:58,435 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:53:58,435 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:53:58,435 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:53:58,435 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/05dd1935f0174433a4c9bcf3fe0dac69, entries=150, sequenceid=594, filesize=11.9 K 2023-04-09 09:53:58,436 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 1732612c92cd83e5f8169e1b3b27bd29 in 537ms, sequenceid=594, compaction requested=true 2023-04-09 09:53:58,436 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1732612c92cd83e5f8169e1b3b27bd29: 2023-04-09 09:53:58,436 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:53:58,436 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:53:58,436 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-09 09:53:58,436 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-09 09:53:58,436 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:53:58,437 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:58,437 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:58,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:58,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:58,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:58,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:58,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:58,437 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:58,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:58,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:58,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:58,438 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,439 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49528 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-09 09:53:58,438 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51968 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-09 09:53:58,439 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): 1732612c92cd83e5f8169e1b3b27bd29/B is initiating minor compaction (all files) 2023-04-09 09:53:58,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:58,439 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1732612c92cd83e5f8169e1b3b27bd29/B in TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:58,439 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1912): 1732612c92cd83e5f8169e1b3b27bd29/A is initiating minor compaction (all files) 2023-04-09 09:53:58,439 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/327f26834c044e32a9d079e451bb7db1, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/d5ce30c0e28b40f4aa1ad2b26d20cc2b, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/3582356e9b9a47eebefb2628826dc5a8, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/a54c08b9597b429bbc533932d98a7702] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp, totalSize=48.4 K 2023-04-09 09:53:58,439 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1732612c92cd83e5f8169e1b3b27bd29/A in TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:58,439 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/a602cab7afbe48cd9969f40505daad0b, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/d8a88c37a5164d61a7ccae323fe3ed2d, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/95c0fd3565da4dacbbf1be2d1b695e35, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/64225f79554c411b9bc55329d68423ab] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp, totalSize=50.8 K 2023-04-09 09:53:58,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:58,440 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 327f26834c044e32a9d079e451bb7db1, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=540, earliestPutTs=1681034035161 2023-04-09 09:53:58,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:58,440 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting a602cab7afbe48cd9969f40505daad0b, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=540, earliestPutTs=1681034035161 2023-04-09 09:53:58,440 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting d5ce30c0e28b40f4aa1ad2b26d20cc2b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=556, earliestPutTs=1681034035806 2023-04-09 09:53:58,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:58,440 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,440 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting d8a88c37a5164d61a7ccae323fe3ed2d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=556, earliestPutTs=1681034035806 2023-04-09 09:53:58,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:58,441 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 3582356e9b9a47eebefb2628826dc5a8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=575, earliestPutTs=1681034036448 2023-04-09 09:53:58,441 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 95c0fd3565da4dacbbf1be2d1b695e35, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=575, earliestPutTs=1681034036448 2023-04-09 09:53:58,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:58,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,442 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting a54c08b9597b429bbc533932d98a7702, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=594, earliestPutTs=1681034037581 2023-04-09 09:53:58,442 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 64225f79554c411b9bc55329d68423ab, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=594, earliestPutTs=1681034037581 2023-04-09 09:53:58,443 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:58,443 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:58,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:58,443 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:58,443 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:58,443 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,443 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:58,443 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:58,443 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:53:58,443 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:53:58,443 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:58,443 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:58,443 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:58,444 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:53:58,444 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:53:58,444 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:53:58,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:58,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:58,457 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): 1732612c92cd83e5f8169e1b3b27bd29#B#compaction#81 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:53:58,462 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1732612c92cd83e5f8169e1b3b27bd29#A#compaction#82 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:53:58,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 1732612c92cd83e5f8169e1b3b27bd29 2023-04-09 09:53:58,520 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1732612c92cd83e5f8169e1b3b27bd29 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-09 09:53:58,521 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=A 2023-04-09 09:53:58,521 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:53:58,521 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=B 2023-04-09 09:53:58,521 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:53:58,521 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=C 2023-04-09 09:53:58,521 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:53:58,544 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:53:58,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:53:58,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:53:58,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:53:58,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:53:58,549 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:53:58,549 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:58,549 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:53:58,549 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:53:58,549 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:58,549 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:58,551 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,552 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:58,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:58,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:58,552 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,552 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:53:58,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:58,552 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:58,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:53:58,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:58,553 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:53:58,555 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:53:58,555 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:53:58,555 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:53:58,555 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:58,555 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:53:58,555 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:53:58,555 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:53:58,555 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:58,556 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. started... 2023-04-09 09:53:58,556 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. as already flushing 2023-04-09 09:53:58,556 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:58,556 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:53:58,556 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:53:58,556 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,556 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:53:58,556 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:53:58,590 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:58,590 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:58,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:58,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:58,594 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:58,594 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:53:58,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:58,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:58,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:58,596 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:53:58,596 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d3178f0[Count = 0] remaining members to acquire global barrier 2023-04-09 09:53:58,596 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:53:58,596 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:58,599 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:58,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39864 deadline: 1681034098590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,599 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:58,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 298 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39920 deadline: 1681034098599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,600 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:58,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39954 deadline: 1681034098600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,629 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:58,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:58,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:58,629 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:53:58,629 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:53:58,629 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:53:58,629 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,629 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:53:58,702 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:58,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39864 deadline: 1681034098700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,703 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:58,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39954 deadline: 1681034098702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,704 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:58,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39920 deadline: 1681034098702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,783 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:53:58,783 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,783 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:58,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,783 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:53:58,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:58,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:58,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:58,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:58,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:58,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:58,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:58,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:53:58,787 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:53:58,787 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:53:58,787 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:53:58,787 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:53:58,787 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:53:58,787 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:53:58,821 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:58,821 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:58,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:58,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:58,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:58,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:58,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:58,821 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:58,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:58,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:58,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:58,822 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:58,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:58,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:58,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:58,823 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:58,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:58,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,867 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,867 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:58,867 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:53:58,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:53:58,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:53:58,867 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:58,867 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:58,868 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:58,868 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:58,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:58,867 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:58,868 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:58,868 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:53:58,868 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:53:58,868 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:58,868 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:58,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:58,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:58,898 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/B/4cb965a085414c5ca8c8647e7ecf7c1a as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/4cb965a085414c5ca8c8647e7ecf7c1a 2023-04-09 09:53:58,901 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/A/987eccf7f23a4a1695a907015ed4cc96 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/987eccf7f23a4a1695a907015ed4cc96 2023-04-09 09:53:58,904 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:58,904 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1732612c92cd83e5f8169e1b3b27bd29/B of 1732612c92cd83e5f8169e1b3b27bd29 into 4cb965a085414c5ca8c8647e7ecf7c1a(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:53:58,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39864 deadline: 1681034098903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,905 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1732612c92cd83e5f8169e1b3b27bd29: 2023-04-09 09:53:58,905 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29., storeName=1732612c92cd83e5f8169e1b3b27bd29/B, priority=12, startTime=1681034038436; duration=0sec 2023-04-09 09:53:58,905 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:53:58,905 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-09 09:53:58,905 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:58,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39920 deadline: 1681034098905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,906 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:58,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39954 deadline: 1681034098905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:58,909 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49528 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-09 09:53:58,909 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): 1732612c92cd83e5f8169e1b3b27bd29/C is initiating minor compaction (all files) 2023-04-09 09:53:58,909 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1732612c92cd83e5f8169e1b3b27bd29/C in TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:58,910 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/c282ada0bf7948f0a1813c115747f468, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/a17924d94fa9457e89574fc641c9ffb4, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/c4a1ea2c4d614a5bad9db3d9bf95b240, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/05dd1935f0174433a4c9bcf3fe0dac69] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp, totalSize=48.4 K 2023-04-09 09:53:58,910 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting c282ada0bf7948f0a1813c115747f468, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=540, earliestPutTs=1681034035161 2023-04-09 09:53:58,910 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting a17924d94fa9457e89574fc641c9ffb4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=556, earliestPutTs=1681034035806 2023-04-09 09:53:58,911 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting c4a1ea2c4d614a5bad9db3d9bf95b240, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=575, earliestPutTs=1681034036448 2023-04-09 09:53:58,911 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 05dd1935f0174433a4c9bcf3fe0dac69, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=594, earliestPutTs=1681034037581 2023-04-09 09:53:58,913 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1732612c92cd83e5f8169e1b3b27bd29/A of 1732612c92cd83e5f8169e1b3b27bd29 into 987eccf7f23a4a1695a907015ed4cc96(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:53:58,913 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1732612c92cd83e5f8169e1b3b27bd29: 2023-04-09 09:53:58,913 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29., storeName=1732612c92cd83e5f8169e1b3b27bd29/A, priority=12, startTime=1681034038436; duration=0sec 2023-04-09 09:53:58,913 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:53:58,920 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): 1732612c92cd83e5f8169e1b3b27bd29#C#compaction#84 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:53:58,933 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=608 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/A/734403ac7e26453b8c80e7a36b14254b 2023-04-09 09:53:58,946 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=608 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/B/e1274c3b707b49d38deadd6855f80477 2023-04-09 09:53:58,968 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:53:58,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:53:58,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:53:58,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:53:58,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:53:58,972 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:53:58,972 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:58,973 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:53:58,973 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:53:58,973 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:58,973 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:59,013 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:59,013 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:59,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:59,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:59,014 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:59,014 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:53:59,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:59,014 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:59,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:53:59,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:59,015 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:53:59,015 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:53:59,015 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:53:59,015 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:53:59,015 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:59,015 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:53:59,015 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:53:59,015 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:53:59,015 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:59,015 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. started... 2023-04-09 09:53:59,016 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. as already flushing 2023-04-09 09:53:59,016 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:59,016 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:53:59,016 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:53:59,016 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:59,016 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:53:59,016 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:53:59,075 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:59,075 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:59,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:59,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:59,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:59,075 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:59,075 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:53:59,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:59,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:59,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:59,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:59,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:59,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:53:59,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1eb5ac57[Count = 0] remaining members to acquire global barrier 2023-04-09 09:53:59,077 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:53:59,077 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:59,113 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:59,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:59,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:59,114 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:53:59,114 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:53:59,114 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:53:59,114 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:59,114 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:53:59,131 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:53:59,131 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:59,131 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:59,131 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:53:59,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:59,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:59,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:59,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:59,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:59,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:59,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:59,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:59,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:59,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:59,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:53:59,134 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:53:59,134 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:53:59,134 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:53:59,134 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:53:59,134 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:53:59,134 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:53:59,147 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:59,147 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:59,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:59,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:59,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:59,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:59,147 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:59,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:59,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:59,147 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:59,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:59,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:59,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:59,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:59,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:59,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:59,149 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:59,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:59,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:59,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:59,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:59,190 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:59,190 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:59,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:53:59,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:59,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:53:59,190 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:59,190 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:59,190 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:59,191 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:59,191 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:53:59,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:59,191 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:53:59,191 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:59,191 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:59,191 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:53:59,191 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:59,191 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:59,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:59,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:59,208 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:59,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39864 deadline: 1681034099207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:59,208 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:59,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39954 deadline: 1681034099208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:59,209 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:59,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 304 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39920 deadline: 1681034099209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:59,291 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:53:59,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:53:59,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:53:59,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:53:59,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:53:59,294 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:53:59,294 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:59,295 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:53:59,295 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:53:59,295 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:59,295 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:59,331 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/C/e1daf8e3c0ff4680a12a25219bd2fac2 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/e1daf8e3c0ff4680a12a25219bd2fac2 2023-04-09 09:53:59,337 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1732612c92cd83e5f8169e1b3b27bd29/C of 1732612c92cd83e5f8169e1b3b27bd29 into e1daf8e3c0ff4680a12a25219bd2fac2(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:53:59,337 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1732612c92cd83e5f8169e1b3b27bd29: 2023-04-09 09:53:59,337 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29., storeName=1732612c92cd83e5f8169e1b3b27bd29/C, priority=12, startTime=1681034038436; duration=0sec 2023-04-09 09:53:59,337 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:53:59,352 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:59,352 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:59,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:59,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:59,352 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:59,352 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:53:59,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:59,353 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:59,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:53:59,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:59,353 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:53:59,353 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:53:59,353 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:53:59,353 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:53:59,354 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:59,354 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:53:59,354 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:53:59,354 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:53:59,354 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:59,354 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. started... 2023-04-09 09:53:59,354 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. as already flushing 2023-04-09 09:53:59,354 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:59,354 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:53:59,354 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:53:59,354 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:59,354 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:53:59,354 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:53:59,373 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=608 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/C/0a860ea9220a4f1ebbb8b0f27f40d966 2023-04-09 09:53:59,379 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/A/734403ac7e26453b8c80e7a36b14254b as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/734403ac7e26453b8c80e7a36b14254b 2023-04-09 09:53:59,384 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/734403ac7e26453b8c80e7a36b14254b, entries=200, sequenceid=608, filesize=14.3 K 2023-04-09 09:53:59,386 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/B/e1274c3b707b49d38deadd6855f80477 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/e1274c3b707b49d38deadd6855f80477 2023-04-09 09:53:59,392 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/e1274c3b707b49d38deadd6855f80477, entries=150, sequenceid=608, filesize=11.9 K 2023-04-09 09:53:59,393 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/C/0a860ea9220a4f1ebbb8b0f27f40d966 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/0a860ea9220a4f1ebbb8b0f27f40d966 2023-04-09 09:53:59,398 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:59,398 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:59,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:59,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:59,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:59,398 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/0a860ea9220a4f1ebbb8b0f27f40d966, entries=150, sequenceid=608, filesize=11.9 K 2023-04-09 09:53:59,398 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:59,399 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:53:59,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:59,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:59,399 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for 1732612c92cd83e5f8169e1b3b27bd29 in 879ms, sequenceid=608, compaction requested=false 2023-04-09 09:53:59,399 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1732612c92cd83e5f8169e1b3b27bd29: 2023-04-09 09:53:59,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:59,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:59,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:59,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:53:59,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@736914e6[Count = 0] remaining members to acquire global barrier 2023-04-09 09:53:59,401 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:53:59,401 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:59,436 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:59,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:59,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:59,437 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:53:59,437 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:53:59,437 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:53:59,437 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:59,437 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:53:59,482 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:53:59,483 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:59,482 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:59,483 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:53:59,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:59,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:59,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:59,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:59,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:59,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:59,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:59,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:59,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:59,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:59,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:53:59,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:53:59,486 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:53:59,486 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:53:59,486 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:53:59,486 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:53:59,486 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:53:59,536 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:59,536 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:59,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:59,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:53:59,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:53:59,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:59,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:59,536 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:59,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:53:59,537 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:59,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:59,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:59,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:59,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:59,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:53:59,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:59,538 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:59,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:59,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:53:59,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:53:59,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:59,567 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:59,567 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:59,567 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:59,567 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:53:59,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:59,567 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:53:59,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:59,567 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:59,567 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:53:59,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:53:59,568 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:59,568 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:59,568 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:53:59,568 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:59,568 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:53:59,568 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:53:59,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:53:59,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:53:59,668 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:53:59,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:53:59,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:53:59,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:53:59,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:53:59,673 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:53:59,673 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:59,673 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:53:59,673 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:53:59,674 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:59,674 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:59,676 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:53:59,676 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:59,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:53:59,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:53:59,676 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:59,676 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:53:59,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:59,677 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:53:59,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:53:59,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:53:59,677 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:53:59,677 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:53:59,677 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:53:59,678 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:53:59,678 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:53:59,678 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:53:59,678 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:53:59,678 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:53:59,678 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:53:59,678 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. started... 2023-04-09 09:53:59,678 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1732612c92cd83e5f8169e1b3b27bd29 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-09 09:53:59,678 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=A 2023-04-09 09:53:59,678 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:53:59,678 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=B 2023-04-09 09:53:59,678 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:53:59,678 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=C 2023-04-09 09:53:59,678 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:53:59,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 1732612c92cd83e5f8169e1b3b27bd29 2023-04-09 09:53:59,712 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. as already flushing 2023-04-09 09:53:59,726 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:59,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39954 deadline: 1681034099723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:59,727 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:59,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39864 deadline: 1681034099724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:59,729 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:59,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 311 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39920 deadline: 1681034099724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:59,831 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:59,831 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:59,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39954 deadline: 1681034099828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:59,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39864 deadline: 1681034099828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:53:59,831 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:53:59,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 313 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39920 deadline: 1681034099831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:00,035 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:00,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39864 deadline: 1681034100032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:00,036 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:00,036 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:00,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 315 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39920 deadline: 1681034100032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:00,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39954 deadline: 1681034100032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:00,088 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=630 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/A/10b43aacc7a7459e91d4d037dd25810f 2023-04-09 09:54:00,106 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=630 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/B/9227135ca1f14ab5953d43331dc8a3fc 2023-04-09 09:54:00,119 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=630 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/C/a9089548e0704963bfc3e0491952b5f6 2023-04-09 09:54:00,125 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/A/10b43aacc7a7459e91d4d037dd25810f as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/10b43aacc7a7459e91d4d037dd25810f 2023-04-09 09:54:00,131 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/10b43aacc7a7459e91d4d037dd25810f, entries=150, sequenceid=630, filesize=11.9 K 2023-04-09 09:54:00,132 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/B/9227135ca1f14ab5953d43331dc8a3fc as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/9227135ca1f14ab5953d43331dc8a3fc 2023-04-09 09:54:00,137 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/9227135ca1f14ab5953d43331dc8a3fc, entries=150, sequenceid=630, filesize=11.9 K 2023-04-09 09:54:00,138 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/C/a9089548e0704963bfc3e0491952b5f6 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/a9089548e0704963bfc3e0491952b5f6 2023-04-09 09:54:00,143 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/a9089548e0704963bfc3e0491952b5f6, entries=150, sequenceid=630, filesize=11.9 K 2023-04-09 09:54:00,144 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 1732612c92cd83e5f8169e1b3b27bd29 in 466ms, sequenceid=630, compaction requested=true 2023-04-09 09:54:00,145 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1732612c92cd83e5f8169e1b3b27bd29: 2023-04-09 09:54:00,145 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:54:00,145 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:00,145 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:00,145 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:00,145 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:00,145 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:00,146 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:00,146 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:00,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:00,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:00,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:00,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:00,147 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:00,147 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:00,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:00,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:00,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:00,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:00,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:00,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6cb4ad1[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:00,149 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:00,149 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:00,150 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:00,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:00,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:00,150 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:00,150 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:00,150 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:00,150 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:00,150 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:00,154 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:00,154 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:00,154 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:00,154 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:00,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:00,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:00,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:00,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:00,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:00,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:00,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:00,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:00,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:00,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:00,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:00,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:00,156 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:00,156 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:00,156 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:00,156 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:00,156 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:00,158 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:00,158 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:00,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:00,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:00,158 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:00,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:00,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:00,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:00,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:00,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:00,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:00,159 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:00,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:00,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:00,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:00,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:00,161 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:00,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:00,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:00,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:00,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:00,167 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:00,167 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:00,167 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:00,167 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:00,167 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:00,167 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:00,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:00,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:00,167 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:00,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:00,167 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:00,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:00,167 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:00,167 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:00,167 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:00,167 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:00,167 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:00,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:00,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:00,267 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:00,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:00,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:00,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:00,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:00,271 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:00,272 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:00,272 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:00,272 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:00,272 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:00,272 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:00,273 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:00,273 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:00,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:00,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:00,274 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:00,274 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:00,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:00,274 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:00,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:00,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:00,275 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:00,275 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:00,275 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:00,275 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:00,275 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:00,275 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:00,276 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:00,276 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:00,276 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:54:00,276 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. started... 2023-04-09 09:54:00,276 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1732612c92cd83e5f8169e1b3b27bd29 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-09 09:54:00,276 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=A 2023-04-09 09:54:00,276 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:00,276 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=B 2023-04-09 09:54:00,276 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:00,276 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=C 2023-04-09 09:54:00,277 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:00,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 1732612c92cd83e5f8169e1b3b27bd29 2023-04-09 09:54:00,339 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. as already flushing 2023-04-09 09:54:00,356 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:00,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39954 deadline: 1681034100353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:00,359 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:00,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 322 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39920 deadline: 1681034100355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:00,360 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:00,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39864 deadline: 1681034100357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:00,459 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:00,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39954 deadline: 1681034100458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:00,462 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:00,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 324 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39920 deadline: 1681034100461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:00,463 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:00,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39864 deadline: 1681034100461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:00,662 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:00,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 303 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39954 deadline: 1681034100661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:00,665 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:00,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 326 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39920 deadline: 1681034100664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:00,665 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:00,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39864 deadline: 1681034100664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:00,690 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=643 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/A/262ec3807c7446d497cae66a1b08f1e1 2023-04-09 09:54:00,965 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:00,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 305 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39954 deadline: 1681034100965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:00,967 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:00,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 328 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39920 deadline: 1681034100967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:00,972 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:00,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39864 deadline: 1681034100968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:01,122 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=643 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/B/7e8678414ea64538b1c1d8c2f6e0e88b 2023-04-09 09:54:01,470 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:01,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 307 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39954 deadline: 1681034101469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:01,471 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:01,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 330 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39920 deadline: 1681034101470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:01,478 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:01,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39864 deadline: 1681034101475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:01,538 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=643 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/C/139c18deb6e94ed88bcc37c83ff0b1bc 2023-04-09 09:54:01,543 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/A/262ec3807c7446d497cae66a1b08f1e1 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/262ec3807c7446d497cae66a1b08f1e1 2023-04-09 09:54:01,548 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/262ec3807c7446d497cae66a1b08f1e1, entries=150, sequenceid=643, filesize=11.9 K 2023-04-09 09:54:01,549 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/B/7e8678414ea64538b1c1d8c2f6e0e88b as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/7e8678414ea64538b1c1d8c2f6e0e88b 2023-04-09 09:54:01,554 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/7e8678414ea64538b1c1d8c2f6e0e88b, entries=150, sequenceid=643, filesize=11.9 K 2023-04-09 09:54:01,555 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/C/139c18deb6e94ed88bcc37c83ff0b1bc as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/139c18deb6e94ed88bcc37c83ff0b1bc 2023-04-09 09:54:01,563 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/139c18deb6e94ed88bcc37c83ff0b1bc, entries=150, sequenceid=643, filesize=11.9 K 2023-04-09 09:54:01,564 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 1732612c92cd83e5f8169e1b3b27bd29 in 1288ms, sequenceid=643, compaction requested=true 2023-04-09 09:54:01,564 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1732612c92cd83e5f8169e1b3b27bd29: 2023-04-09 09:54:01,564 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:54:01,564 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:01,565 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:01,565 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:01,568 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:01,568 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:01,569 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:01,569 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:01,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:01,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:01,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:01,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:01,570 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:01,570 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:01,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:01,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:01,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:01,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:01,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:01,572 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:01,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@201b5400[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:01,572 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:01,574 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:01,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:01,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:01,574 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:01,574 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:01,574 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:01,574 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:01,575 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:01,576 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:01,576 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:01,576 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:01,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:01,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:01,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:01,576 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:01,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:01,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:01,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:01,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:01,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:01,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:01,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:01,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:01,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:01,579 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:01,579 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:01,579 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:01,579 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:01,579 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:01,580 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:01,580 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:01,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:01,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:01,581 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:01,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:01,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:01,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:01,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:01,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:01,581 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:01,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:01,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:01,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:01,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:01,583 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:01,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:01,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:01,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:01,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:01,585 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:01,585 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:01,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:01,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:01,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:01,585 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:01,585 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:01,585 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:01,585 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:01,585 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:01,585 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:01,585 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:01,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:01,585 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:01,585 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:01,586 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:01,586 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:01,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:01,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:01,686 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:01,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:01,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:01,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:01,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:01,690 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:01,690 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:01,690 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:01,690 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:01,690 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:01,690 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:01,693 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:01,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:01,693 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:01,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:01,693 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:01,693 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:01,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:01,694 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:01,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:01,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:01,694 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:01,694 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:01,695 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:01,695 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:01,695 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:01,695 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:01,716 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:01,716 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:54:01,716 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. started... 2023-04-09 09:54:01,716 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:01,716 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1732612c92cd83e5f8169e1b3b27bd29 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-09 09:54:01,716 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=A 2023-04-09 09:54:01,717 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:01,717 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=B 2023-04-09 09:54:01,717 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:01,717 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=C 2023-04-09 09:54:01,717 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:01,728 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=662 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/A/e91aea20cd8e41bd922966345330db5b 2023-04-09 09:54:01,746 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=662 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/B/3f63ff6b4ecd4df4bd660d2cf1cc4bb6 2023-04-09 09:54:01,766 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=662 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/C/c5cbd7daed4e475a8f61adb4cdbd21a0 2023-04-09 09:54:01,771 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/A/e91aea20cd8e41bd922966345330db5b as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/e91aea20cd8e41bd922966345330db5b 2023-04-09 09:54:01,776 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/e91aea20cd8e41bd922966345330db5b, entries=150, sequenceid=662, filesize=11.9 K 2023-04-09 09:54:01,777 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/B/3f63ff6b4ecd4df4bd660d2cf1cc4bb6 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/3f63ff6b4ecd4df4bd660d2cf1cc4bb6 2023-04-09 09:54:01,784 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/3f63ff6b4ecd4df4bd660d2cf1cc4bb6, entries=150, sequenceid=662, filesize=11.9 K 2023-04-09 09:54:01,786 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/C/c5cbd7daed4e475a8f61adb4cdbd21a0 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/c5cbd7daed4e475a8f61adb4cdbd21a0 2023-04-09 09:54:01,791 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/c5cbd7daed4e475a8f61adb4cdbd21a0, entries=150, sequenceid=662, filesize=11.9 K 2023-04-09 09:54:01,792 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=0 B/0 for 1732612c92cd83e5f8169e1b3b27bd29 in 76ms, sequenceid=662, compaction requested=true 2023-04-09 09:54:01,792 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1732612c92cd83e5f8169e1b3b27bd29: 2023-04-09 09:54:01,792 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:54:01,792 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:01,792 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:01,792 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:01,796 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:01,796 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:01,797 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:01,797 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:01,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:01,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:01,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:01,798 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:01,798 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:01,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:01,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:01,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:01,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:01,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:01,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:01,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@137f8340[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:01,801 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:01,801 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:01,802 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:01,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:01,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:01,802 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:01,802 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:01,802 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:01,802 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:01,802 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:01,803 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:01,804 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:01,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:01,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:01,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:01,804 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:01,804 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:01,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:01,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:01,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:01,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:01,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:01,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:01,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:01,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:01,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:01,814 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:01,815 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:01,815 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:01,815 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:01,815 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:01,817 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:01,817 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:01,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:01,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:01,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:01,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:01,817 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:01,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:01,818 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:01,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:01,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:01,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:01,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:01,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:01,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:01,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:01,819 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:01,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:01,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:01,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:01,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:01,822 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:01,822 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:01,822 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:01,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:01,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:01,822 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:01,822 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:01,822 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:01,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:01,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:01,823 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:01,823 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:01,823 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:01,823 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:01,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:01,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:01,823 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:01,823 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:01,824 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:01,924 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:01,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:01,926 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:01,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:01,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:01,927 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:01,927 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:01,927 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:01,927 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:01,928 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:01,928 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:01,929 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:01,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:01,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:01,929 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:01,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:01,930 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:01,930 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:01,930 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:01,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:01,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:01,930 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:01,931 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:01,931 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:01,931 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:01,931 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:01,931 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:01,931 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:01,931 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:01,931 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:54:01,931 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. started... 2023-04-09 09:54:01,931 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1732612c92cd83e5f8169e1b3b27bd29: 2023-04-09 09:54:01,931 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:54:01,931 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:01,931 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:01,931 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:01,931 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:01,931 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:01,933 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:01,933 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:01,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:01,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:01,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:01,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:01,933 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:01,933 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:01,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:01,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:01,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:01,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:01,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:01,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@280a6bed[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:01,935 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:01,935 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:01,937 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:01,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:01,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:01,937 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:01,937 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:01,937 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:01,937 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:01,937 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:01,940 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:01,940 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:01,940 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:01,940 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:01,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:01,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:01,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:01,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:01,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:01,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:01,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:01,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:01,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:01,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:01,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:01,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:01,943 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:01,943 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:01,943 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:01,943 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:01,943 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:01,945 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:01,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:01,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:01,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:01,945 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:01,945 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:01,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:01,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:01,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:01,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:01,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:01,946 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:01,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:01,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:01,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:01,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:01,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:01,947 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:01,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:01,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:01,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:01,953 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:01,953 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:01,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:01,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:01,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:01,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:01,954 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:01,954 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:01,954 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:01,954 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:01,954 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:01,954 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:01,954 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:01,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:01,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:01,954 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:01,954 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:01,954 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:01,954 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:02,054 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:02,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:02,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:02,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:02,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:02,064 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:02,064 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:02,064 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:02,064 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:02,065 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:02,065 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:02,066 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:02,066 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:02,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:02,066 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,067 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:02,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:02,067 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:02,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:02,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:02,067 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:02,068 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:02,068 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:02,068 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:02,068 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:02,068 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:02,068 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:02,068 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:02,068 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:54:02,068 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. started... 2023-04-09 09:54:02,069 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1732612c92cd83e5f8169e1b3b27bd29: 2023-04-09 09:54:02,069 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:54:02,069 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:02,069 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:02,069 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,069 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:02,069 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:02,070 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,070 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:02,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:02,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:02,071 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:02,071 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:02,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:02,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:02,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:02,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:02,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:02,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@744b21ff[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:02,072 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:02,073 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:02,074 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:02,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:02,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:02,075 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:02,075 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:02,075 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:02,075 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,075 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:02,077 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:02,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:02,077 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:02,077 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:02,077 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:02,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:02,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:02,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:02,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:02,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:02,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:02,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:02,080 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:02,080 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:02,080 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:02,080 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:02,080 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:02,082 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:02,082 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:02,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:02,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:02,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:02,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:02,082 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:02,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:02,082 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:02,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:02,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:02,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:02,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:02,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:02,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:02,083 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:02,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:02,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,085 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:02,085 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:02,085 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:02,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:02,085 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:02,085 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:02,086 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,086 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:02,086 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:02,086 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:02,086 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:02,086 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:02,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:02,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:02,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:02,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:02,086 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:02,086 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:02,186 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:02,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:02,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:02,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:02,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:02,190 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:02,190 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:02,190 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:02,190 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:02,192 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:02,192 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:02,193 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,194 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,194 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:02,196 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:02,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:02,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:02,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:02,197 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:02,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:02,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:02,197 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:02,197 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:02,197 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:02,198 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:02,198 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:02,198 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:02,198 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:02,198 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:02,198 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:54:02,198 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. started... 2023-04-09 09:54:02,198 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1732612c92cd83e5f8169e1b3b27bd29: 2023-04-09 09:54:02,198 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:54:02,198 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:02,198 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:02,198 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,198 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:02,198 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:02,203 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,203 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:02,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:02,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:02,204 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:02,204 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:02,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:02,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:02,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:02,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:02,206 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:02,206 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:02,206 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e8ac022[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:02,206 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:02,207 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:02,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:02,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:02,207 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:02,207 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:02,207 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:02,208 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,208 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:02,209 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,209 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:02,209 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:02,209 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:02,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:02,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:02,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:02,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:02,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:02,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:02,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:02,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:02,213 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:02,213 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:02,214 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:02,214 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:02,214 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:02,214 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:02,215 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:02,215 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:02,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:02,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:02,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:02,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:02,215 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:02,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:02,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:02,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:02,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:02,216 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:02,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:02,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:02,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:02,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,218 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:02,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:02,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,221 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,222 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:02,222 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:02,222 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:02,222 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:02,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:02,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:02,222 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:02,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:02,222 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:02,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:02,222 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,222 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:02,222 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:02,222 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:02,222 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:02,222 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:02,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:02,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:02,322 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:02,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:02,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:02,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:02,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:02,325 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:02,326 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:02,326 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:02,326 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:02,326 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:02,326 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:02,327 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,327 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:02,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:02,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:02,328 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,328 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:02,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:02,328 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:02,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:02,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:02,329 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:02,329 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:02,329 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:02,329 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:02,329 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:02,329 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:02,329 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:02,329 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:02,329 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:54:02,329 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. started... 2023-04-09 09:54:02,330 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1732612c92cd83e5f8169e1b3b27bd29: 2023-04-09 09:54:02,330 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:54:02,330 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:02,330 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:02,330 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,330 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:02,330 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:02,331 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,331 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:02,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:02,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:02,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:02,332 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:02,332 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:02,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:02,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:02,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:02,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:02,333 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:02,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61002730[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:02,333 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:02,336 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:02,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:02,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:02,336 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:02,336 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:02,336 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,336 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:02,336 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:02,338 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,338 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:02,338 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:02,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:02,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:02,338 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:02,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:02,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:02,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:02,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:02,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:02,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:02,340 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:02,340 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:02,340 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:02,340 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:02,340 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:02,340 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:02,342 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:02,342 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:02,342 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:02,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:02,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:02,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:02,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:02,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:02,342 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:02,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:02,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:02,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:02,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:02,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:02,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:02,344 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:02,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:02,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,346 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,346 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:02,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:02,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:02,346 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:02,346 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:02,346 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,346 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:02,346 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:02,346 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:02,346 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:02,346 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:02,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:02,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:02,346 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:02,346 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:02,346 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:02,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:02,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:02,447 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:02,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:02,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:02,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:02,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:02,451 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:02,451 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:02,451 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:02,451 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:02,451 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:02,451 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:02,457 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:02,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:02,457 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:02,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:02,457 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,457 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:02,458 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:02,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:02,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:02,458 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:02,458 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:02,459 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:02,459 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:02,459 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:02,459 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:02,459 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:02,459 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:02,459 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:54:02,459 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. started... 2023-04-09 09:54:02,459 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1732612c92cd83e5f8169e1b3b27bd29: 2023-04-09 09:54:02,459 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:54:02,459 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:02,459 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:02,459 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,459 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:02,459 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:02,461 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,461 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:02,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:02,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:02,461 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:02,461 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:02,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:02,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:02,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:02,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:02,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:02,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1aae6dfe[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:02,464 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:02,464 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:02,466 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:02,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:02,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:02,466 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:02,466 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:02,466 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:02,466 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,466 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:02,468 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:02,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:02,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:02,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:02,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:02,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:02,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:02,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:02,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:02,471 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:02,472 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:02,472 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:02,472 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:02,472 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:02,473 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:02,473 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:02,473 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:02,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:02,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:02,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:02,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:02,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:02,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:02,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:02,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:02,475 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:02,475 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:02,475 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:02,476 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:02,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:02,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:02,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:02,478 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:02,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:02,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,482 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,482 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:02,482 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:02,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:02,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:02,482 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:02,482 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:02,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:02,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:02,482 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:02,482 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,482 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:02,482 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:02,482 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:02,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:02,482 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:02,482 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:02,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:02,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 1732612c92cd83e5f8169e1b3b27bd29 2023-04-09 09:54:02,490 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1732612c92cd83e5f8169e1b3b27bd29 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-09 09:54:02,490 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=A 2023-04-09 09:54:02,490 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:02,490 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=B 2023-04-09 09:54:02,490 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:02,490 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=C 2023-04-09 09:54:02,490 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:02,510 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=677 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/A/0893e2987ebf414c99553f57de4fa912 2023-04-09 09:54:02,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681034042502, max timestamp=1681034042515Num uniques 859; 2023-04-09 09:54:02,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-09 09:54:02,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681034042502, max timestamp=1681034042515Num uniques 859; 2023-04-09 09:54:02,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-09 09:54:02,519 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681034042502, max timestamp=1681034042515Num uniques 859; 2023-04-09 09:54:02,519 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-09 09:54:02,545 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:02,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 345 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39920 deadline: 1681034102544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,547 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=677 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/B/106807961ad643659d495644d89f1125 2023-04-09 09:54:02,550 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:02,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 323 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39954 deadline: 1681034102544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,553 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:02,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39864 deadline: 1681034102545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,583 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:02,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:02,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:02,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:02,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:02,592 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:02,592 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:02,592 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:02,592 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:02,593 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:02,593 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:02,594 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:02,594 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:02,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:02,595 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:02,595 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:02,595 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:02,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:02,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:02,596 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:02,596 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:02,596 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:02,596 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:02,596 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:02,596 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:02,599 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:02,599 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:02,599 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:54:02,599 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. started... 2023-04-09 09:54:02,599 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. as already flushing 2023-04-09 09:54:02,600 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:54:02,600 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:02,600 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:02,600 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,600 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:02,600 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:02,601 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:02,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:02,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:02,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:02,602 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:02,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:02,603 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:02,603 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:02,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:02,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:02,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37c7f38d[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:02,604 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:02,604 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:02,605 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:02,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:02,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:02,605 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:02,605 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,605 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:02,605 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:02,605 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:02,608 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:02,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:02,608 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:02,608 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:02,608 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:02,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:02,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:02,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:02,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:02,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:02,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:02,613 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:02,613 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:02,613 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:02,613 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:02,613 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:02,613 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:02,615 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:02,615 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:02,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:02,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:02,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:02,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:02,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:02,615 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:02,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:02,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:02,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:02,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:02,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:02,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:02,620 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:02,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,626 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:02,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:02,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,629 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:02,629 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:02,629 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:02,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:02,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:02,629 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:02,629 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:02,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:02,629 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:02,630 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:02,630 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,630 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:02,630 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:02,630 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:02,629 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:02,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:02,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:02,648 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:02,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 347 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39920 deadline: 1681034102647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,658 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:02,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 325 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39954 deadline: 1681034102654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,658 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:02,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39864 deadline: 1681034102654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,729 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:02,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:02,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:02,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:02,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:02,734 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:02,734 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:02,734 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:02,734 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:02,735 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:02,735 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:02,738 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:02,738 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:02,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:02,739 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,739 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:02,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:02,739 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:02,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:02,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:02,740 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:02,753 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:02,753 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:02,753 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:02,753 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:02,753 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:02,754 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:02,754 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:02,754 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:54:02,754 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. started... 2023-04-09 09:54:02,754 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. as already flushing 2023-04-09 09:54:02,754 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:54:02,754 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:02,754 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:02,754 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,754 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:02,754 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:02,756 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,756 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:02,757 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:02,757 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:02,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:02,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:02,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:02,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:02,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:02,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:02,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:02,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f14ae87[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:02,760 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:02,760 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:02,762 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:02,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:02,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:02,762 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:02,762 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:02,762 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:02,764 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,764 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:02,765 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,765 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:02,765 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:02,765 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:02,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:02,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:02,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:02,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:02,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:02,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:02,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:02,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:02,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:02,770 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:02,770 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:02,770 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:02,770 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:02,770 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:02,773 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:02,773 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:02,773 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:02,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:02,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:02,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:02,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:02,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:02,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:02,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:02,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:02,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:02,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:02,774 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:02,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:02,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:02,777 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:02,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,781 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:02,781 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:02,781 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,781 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:02,781 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:02,781 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,781 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:02,781 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:02,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:02,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:02,781 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:02,781 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:02,781 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:02,781 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:02,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:02,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:02,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:02,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:02,850 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:02,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 349 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39920 deadline: 1681034102849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,865 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:02,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 327 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39954 deadline: 1681034102860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,866 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:02,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39864 deadline: 1681034102863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,882 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:02,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:02,884 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:02,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:02,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:02,885 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:02,885 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:02,885 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:02,885 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:02,886 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:02,886 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:02,887 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:02,887 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:02,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:02,887 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,887 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:02,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:02,888 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:02,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:02,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:02,888 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:02,888 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:02,888 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:02,888 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:02,888 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:02,888 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:02,889 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:02,889 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:02,889 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:54:02,889 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. started... 2023-04-09 09:54:02,889 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. as already flushing 2023-04-09 09:54:02,889 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:54:02,889 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:02,889 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:02,889 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,889 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:02,889 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:02,890 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,890 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:02,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:02,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:02,891 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:02,891 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:02,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:02,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:02,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:02,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:02,892 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:02,892 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@fcd3df2[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:02,892 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:02,892 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:02,897 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:02,897 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,897 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:02,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:02,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:02,897 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:02,897 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:02,897 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:02,898 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:02,898 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:02,898 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,898 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:02,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:02,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:02,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:02,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:02,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:02,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:02,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:02,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:02,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:02,902 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:02,903 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:02,903 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:02,903 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:02,903 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:02,904 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:02,904 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:02,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:02,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:02,904 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:02,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:02,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:02,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:02,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:02,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:02,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:02,905 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:02,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:02,906 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:02,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:02,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:02,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:02,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,912 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,912 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:02,912 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:02,912 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:02,912 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:02,912 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:02,912 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:02,912 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:02,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:02,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:02,912 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:02,912 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:02,912 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:02,912 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:02,912 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:02,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:02,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:02,912 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:02,912 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:02,971 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=677 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/C/ae4929ca81fc459080b9b6b3b9882e33 2023-04-09 09:54:02,978 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/A/0893e2987ebf414c99553f57de4fa912 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/0893e2987ebf414c99553f57de4fa912 2023-04-09 09:54:02,984 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/0893e2987ebf414c99553f57de4fa912, entries=250, sequenceid=677, filesize=16.7 K 2023-04-09 09:54:02,985 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/B/106807961ad643659d495644d89f1125 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/106807961ad643659d495644d89f1125 2023-04-09 09:54:02,990 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/106807961ad643659d495644d89f1125, entries=100, sequenceid=677, filesize=9.5 K 2023-04-09 09:54:02,991 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/C/ae4929ca81fc459080b9b6b3b9882e33 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/ae4929ca81fc459080b9b6b3b9882e33 2023-04-09 09:54:02,998 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/ae4929ca81fc459080b9b6b3b9882e33, entries=100, sequenceid=677, filesize=9.5 K 2023-04-09 09:54:02,999 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=136.42 KB/139690 for 1732612c92cd83e5f8169e1b3b27bd29 in 509ms, sequenceid=677, compaction requested=true 2023-04-09 09:54:02,999 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1732612c92cd83e5f8169e1b3b27bd29: 2023-04-09 09:54:02,999 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:54:02,999 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-09 09:54:02,999 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-09 09:54:02,999 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:54:02,999 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:54:03,002 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 81352 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-09 09:54:03,002 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71592 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-09 09:54:03,002 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1912): 1732612c92cd83e5f8169e1b3b27bd29/B is initiating minor compaction (all files) 2023-04-09 09:54:03,002 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1732612c92cd83e5f8169e1b3b27bd29/B in TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:54:03,002 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/4cb965a085414c5ca8c8647e7ecf7c1a, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/e1274c3b707b49d38deadd6855f80477, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/9227135ca1f14ab5953d43331dc8a3fc, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/7e8678414ea64538b1c1d8c2f6e0e88b, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/3f63ff6b4ecd4df4bd660d2cf1cc4bb6, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/106807961ad643659d495644d89f1125] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp, totalSize=69.9 K 2023-04-09 09:54:03,002 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): 1732612c92cd83e5f8169e1b3b27bd29/A is initiating minor compaction (all files) 2023-04-09 09:54:03,002 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1732612c92cd83e5f8169e1b3b27bd29/A in TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:54:03,003 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/987eccf7f23a4a1695a907015ed4cc96, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/734403ac7e26453b8c80e7a36b14254b, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/10b43aacc7a7459e91d4d037dd25810f, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/262ec3807c7446d497cae66a1b08f1e1, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/e91aea20cd8e41bd922966345330db5b, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/0893e2987ebf414c99553f57de4fa912] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp, totalSize=79.4 K 2023-04-09 09:54:03,003 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 4cb965a085414c5ca8c8647e7ecf7c1a, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=594, earliestPutTs=1681034037581 2023-04-09 09:54:03,003 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 987eccf7f23a4a1695a907015ed4cc96, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=594, earliestPutTs=1681034037581 2023-04-09 09:54:03,003 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting e1274c3b707b49d38deadd6855f80477, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=608, earliestPutTs=1681034037915 2023-04-09 09:54:03,004 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 734403ac7e26453b8c80e7a36b14254b, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=608, earliestPutTs=1681034037912 2023-04-09 09:54:03,004 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 9227135ca1f14ab5953d43331dc8a3fc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=630, earliestPutTs=1681034038578 2023-04-09 09:54:03,004 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 10b43aacc7a7459e91d4d037dd25810f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=630, earliestPutTs=1681034038578 2023-04-09 09:54:03,004 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 7e8678414ea64538b1c1d8c2f6e0e88b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=643, earliestPutTs=1681034039720 2023-04-09 09:54:03,005 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 262ec3807c7446d497cae66a1b08f1e1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=643, earliestPutTs=1681034039720 2023-04-09 09:54:03,005 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 3f63ff6b4ecd4df4bd660d2cf1cc4bb6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=662, earliestPutTs=1681034040352 2023-04-09 09:54:03,005 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting e91aea20cd8e41bd922966345330db5b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=662, earliestPutTs=1681034040352 2023-04-09 09:54:03,005 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 106807961ad643659d495644d89f1125, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=677, earliestPutTs=1681034042484 2023-04-09 09:54:03,006 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 0893e2987ebf414c99553f57de4fa912, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=677, earliestPutTs=1681034042477 2023-04-09 09:54:03,012 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:03,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:03,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:03,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:03,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:03,016 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:03,016 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:03,016 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:03,016 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:03,017 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:03,017 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:03,019 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:03,020 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:03,020 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:03,020 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:03,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:03,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:03,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:03,020 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:03,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:03,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:03,021 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:03,022 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1732612c92cd83e5f8169e1b3b27bd29#B#compaction#100 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:54:03,023 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): 1732612c92cd83e5f8169e1b3b27bd29#A#compaction#99 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:54:03,028 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:03,028 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:03,028 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:03,028 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:03,028 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:03,029 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:03,029 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:03,029 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:54:03,029 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. started... 2023-04-09 09:54:03,029 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1732612c92cd83e5f8169e1b3b27bd29 3/3 column families, dataSize=136.42 KB heapSize=380.69 KB 2023-04-09 09:54:03,029 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=A 2023-04-09 09:54:03,029 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:54:03,030 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=B 2023-04-09 09:54:03,030 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:54:03,030 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=C 2023-04-09 09:54:03,030 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:54:03,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 1732612c92cd83e5f8169e1b3b27bd29 2023-04-09 09:54:03,156 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. as already flushing 2023-04-09 09:54:03,176 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:03,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 330 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39954 deadline: 1681034103170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:03,179 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:03,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 356 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39920 deadline: 1681034103175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:03,179 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:03,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39864 deadline: 1681034103176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:03,279 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:03,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 332 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39954 deadline: 1681034103277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:03,284 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:03,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 358 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39920 deadline: 1681034103280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:03,284 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:03,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 287 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39864 deadline: 1681034103281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:03,449 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/A/842c7182675a41bd992dd2c868722d00 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/842c7182675a41bd992dd2c868722d00 2023-04-09 09:54:03,453 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/B/6957a78ab87a431fad086a7395f5eb94 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/6957a78ab87a431fad086a7395f5eb94 2023-04-09 09:54:03,461 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=712 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/A/14ef4962bdb1439ea4d02653ab3b65f1 2023-04-09 09:54:03,462 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 1732612c92cd83e5f8169e1b3b27bd29/B of 1732612c92cd83e5f8169e1b3b27bd29 into 6957a78ab87a431fad086a7395f5eb94(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:54:03,462 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1732612c92cd83e5f8169e1b3b27bd29: 2023-04-09 09:54:03,462 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29., storeName=1732612c92cd83e5f8169e1b3b27bd29/B, priority=10, startTime=1681034042999; duration=0sec 2023-04-09 09:54:03,462 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:54:03,462 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 1732612c92cd83e5f8169e1b3b27bd29/A of 1732612c92cd83e5f8169e1b3b27bd29 into 842c7182675a41bd992dd2c868722d00(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:54:03,462 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-09 09:54:03,465 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1732612c92cd83e5f8169e1b3b27bd29: 2023-04-09 09:54:03,465 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29., storeName=1732612c92cd83e5f8169e1b3b27bd29/A, priority=10, startTime=1681034042999; duration=0sec 2023-04-09 09:54:03,466 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:54:03,467 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71592 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-09 09:54:03,467 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1912): 1732612c92cd83e5f8169e1b3b27bd29/C is initiating minor compaction (all files) 2023-04-09 09:54:03,467 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1732612c92cd83e5f8169e1b3b27bd29/C in TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:54:03,468 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/e1daf8e3c0ff4680a12a25219bd2fac2, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/0a860ea9220a4f1ebbb8b0f27f40d966, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/a9089548e0704963bfc3e0491952b5f6, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/139c18deb6e94ed88bcc37c83ff0b1bc, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/c5cbd7daed4e475a8f61adb4cdbd21a0, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/ae4929ca81fc459080b9b6b3b9882e33] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp, totalSize=69.9 K 2023-04-09 09:54:03,468 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting e1daf8e3c0ff4680a12a25219bd2fac2, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=594, earliestPutTs=1681034037581 2023-04-09 09:54:03,469 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 0a860ea9220a4f1ebbb8b0f27f40d966, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=608, earliestPutTs=1681034037915 2023-04-09 09:54:03,469 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting a9089548e0704963bfc3e0491952b5f6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=630, earliestPutTs=1681034038578 2023-04-09 09:54:03,469 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 139c18deb6e94ed88bcc37c83ff0b1bc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=643, earliestPutTs=1681034039720 2023-04-09 09:54:03,470 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting c5cbd7daed4e475a8f61adb4cdbd21a0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=662, earliestPutTs=1681034040352 2023-04-09 09:54:03,470 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting ae4929ca81fc459080b9b6b3b9882e33, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=677, earliestPutTs=1681034042484 2023-04-09 09:54:03,485 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:03,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 334 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39954 deadline: 1681034103482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:03,488 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:03,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 360 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39920 deadline: 1681034103485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:03,489 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:03,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 289 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39864 deadline: 1681034103486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:03,498 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1732612c92cd83e5f8169e1b3b27bd29#C#compaction#102 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:54:03,531 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/C/5c3836131ca7494e9dec0068135ed226 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/5c3836131ca7494e9dec0068135ed226 2023-04-09 09:54:03,542 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 1732612c92cd83e5f8169e1b3b27bd29/C of 1732612c92cd83e5f8169e1b3b27bd29 into 5c3836131ca7494e9dec0068135ed226(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:54:03,542 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1732612c92cd83e5f8169e1b3b27bd29: 2023-04-09 09:54:03,542 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29., storeName=1732612c92cd83e5f8169e1b3b27bd29/C, priority=10, startTime=1681034042999; duration=0sec 2023-04-09 09:54:03,543 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:54:03,789 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:03,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 336 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39954 deadline: 1681034103789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:03,794 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:03,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 291 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39864 deadline: 1681034103790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:03,797 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:03,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 362 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39920 deadline: 1681034103792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:03,947 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=712 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/B/4ab0f8ad0a774c65bd2427f47c55b986 2023-04-09 09:54:04,297 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:04,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 338 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39954 deadline: 1681034104294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:04,297 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:04,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 293 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39864 deadline: 1681034104295, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:04,303 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:04,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 364 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39920 deadline: 1681034104300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:04,360 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=712 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/C/196a5464b72548daa651189948fdd7f4 2023-04-09 09:54:04,366 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/A/14ef4962bdb1439ea4d02653ab3b65f1 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/14ef4962bdb1439ea4d02653ab3b65f1 2023-04-09 09:54:04,376 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/14ef4962bdb1439ea4d02653ab3b65f1, entries=150, sequenceid=712, filesize=11.9 K 2023-04-09 09:54:04,379 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/B/4ab0f8ad0a774c65bd2427f47c55b986 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/4ab0f8ad0a774c65bd2427f47c55b986 2023-04-09 09:54:04,384 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/4ab0f8ad0a774c65bd2427f47c55b986, entries=150, sequenceid=712, filesize=11.9 K 2023-04-09 09:54:04,385 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/C/196a5464b72548daa651189948fdd7f4 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/196a5464b72548daa651189948fdd7f4 2023-04-09 09:54:04,389 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/196a5464b72548daa651189948fdd7f4, entries=150, sequenceid=712, filesize=11.9 K 2023-04-09 09:54:04,390 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~136.42 KB/139690, heapSize ~380.64 KB/389776, currentSize=46.96 KB/48090 for 1732612c92cd83e5f8169e1b3b27bd29 in 1361ms, sequenceid=712, compaction requested=false 2023-04-09 09:54:04,390 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1732612c92cd83e5f8169e1b3b27bd29: 2023-04-09 09:54:04,390 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:54:04,390 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:04,390 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:04,390 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:04,390 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:04,390 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:04,393 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:04,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:04,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:04,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:04,394 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:04,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:04,394 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:04,394 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:04,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:04,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:04,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:04,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:04,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:04,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66657068[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:04,396 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:04,396 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:04,397 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:04,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:04,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:04,397 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:04,397 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:04,397 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:04,397 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:04,397 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:04,398 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:04,398 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:04,398 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:04,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:04,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:04,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:04,399 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:04,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:04,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:04,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:04,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:04,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:04,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:04,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:04,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:04,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:04,402 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:04,402 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:04,402 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:04,403 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:04,403 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:04,405 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:04,405 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:04,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:04,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:04,405 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:04,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:04,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:04,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:04,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:04,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:04,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:04,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:04,408 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:04,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:04,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:04,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:04,410 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:04,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:04,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:04,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:04,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:04,413 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:04,413 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:04,413 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:04,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:04,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:04,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:04,413 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:04,413 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:04,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:04,413 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:04,413 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:04,413 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:04,414 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:04,414 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:04,414 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:04,414 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:04,414 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:04,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:04,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:04,514 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:04,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:04,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:04,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:04,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:04,518 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:04,518 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:04,518 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:04,518 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:04,518 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:04,518 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:04,520 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:04,520 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:04,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:04,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:04,520 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:04,520 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:04,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:04,520 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:04,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:04,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:04,521 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:04,522 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:04,522 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:04,522 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:04,522 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:04,522 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:04,522 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:04,522 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:04,522 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:54:04,522 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. started... 2023-04-09 09:54:04,522 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1732612c92cd83e5f8169e1b3b27bd29 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-09 09:54:04,522 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=A 2023-04-09 09:54:04,523 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:04,523 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=B 2023-04-09 09:54:04,523 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:04,523 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=C 2023-04-09 09:54:04,523 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:04,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 1732612c92cd83e5f8169e1b3b27bd29 2023-04-09 09:54:04,533 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. as already flushing 2023-04-09 09:54:04,572 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681034044557, max timestamp=1681034044570Num uniques 859; 2023-04-09 09:54:04,572 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681034044557, max timestamp=1681034044570Num uniques 859; 2023-04-09 09:54:04,572 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-09 09:54:04,572 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681034044557, max timestamp=1681034044570Num uniques 859; 2023-04-09 09:54:04,572 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-09 09:54:04,572 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-09 09:54:04,594 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:04,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39926 deadline: 1681034104593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:04,595 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:04,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39904 deadline: 1681034104594, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:04,640 DEBUG [Thread-154] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x49a45b6d to 127.0.0.1:57816 2023-04-09 09:54:04,640 DEBUG [Thread-154] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-09 09:54:04,641 DEBUG [Thread-158] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3cd3abbd to 127.0.0.1:57816 2023-04-09 09:54:04,641 DEBUG [Thread-158] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-09 09:54:04,643 DEBUG [Thread-152] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0dbb9ac1 to 127.0.0.1:57816 2023-04-09 09:54:04,643 DEBUG [Thread-152] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-09 09:54:04,647 DEBUG [Thread-156] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x04fb2b82 to 127.0.0.1:57816 2023-04-09 09:54:04,647 DEBUG [Thread-156] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-09 09:54:04,696 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:04,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39926 deadline: 1681034104696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:04,697 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:04,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39904 deadline: 1681034104697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:04,898 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:04,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39926 deadline: 1681034104898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:04,900 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:04,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39904 deadline: 1681034104900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:04,942 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=725 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/A/05ff574f39424b92b0babf4c85a6e574 2023-04-09 09:54:05,199 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:05,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39926 deadline: 1681034105199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:05,203 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:05,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39904 deadline: 1681034105203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:05,303 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:05,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 295 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39864 deadline: 1681034105302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:05,305 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:05,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 340 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39954 deadline: 1681034105305, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:05,312 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:05,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 366 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39920 deadline: 1681034105312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:05,365 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=725 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/B/14582e1241b54e5283fc3d084ea4d525 2023-04-09 09:54:05,703 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:05,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39926 deadline: 1681034105703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:05,705 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:05,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39904 deadline: 1681034105705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:05,776 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=725 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/C/1e0a5a5c96a14601b47e0f63addedf1d 2023-04-09 09:54:05,781 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/A/05ff574f39424b92b0babf4c85a6e574 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/05ff574f39424b92b0babf4c85a6e574 2023-04-09 09:54:05,785 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/05ff574f39424b92b0babf4c85a6e574, entries=150, sequenceid=725, filesize=11.9 K 2023-04-09 09:54:05,786 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/B/14582e1241b54e5283fc3d084ea4d525 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/14582e1241b54e5283fc3d084ea4d525 2023-04-09 09:54:05,790 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/14582e1241b54e5283fc3d084ea4d525, entries=150, sequenceid=725, filesize=11.9 K 2023-04-09 09:54:05,791 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/C/1e0a5a5c96a14601b47e0f63addedf1d as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/1e0a5a5c96a14601b47e0f63addedf1d 2023-04-09 09:54:05,795 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/1e0a5a5c96a14601b47e0f63addedf1d, entries=150, sequenceid=725, filesize=11.9 K 2023-04-09 09:54:05,796 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 1732612c92cd83e5f8169e1b3b27bd29 in 1274ms, sequenceid=725, compaction requested=true 2023-04-09 09:54:05,796 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1732612c92cd83e5f8169e1b3b27bd29: 2023-04-09 09:54:05,796 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:54:05,796 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:05,796 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:05,797 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:05,797 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:05,797 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:05,798 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:05,798 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:05,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:05,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:05,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:05,798 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:05,798 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:05,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:05,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:05,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:05,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:05,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:05,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:05,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e36acb2[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:05,800 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:05,800 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:05,801 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:05,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:05,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:05,801 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:05,801 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:05,801 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:05,801 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:05,801 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:05,804 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:05,804 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:05,804 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:05,804 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:05,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:05,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:05,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:05,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:05,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:05,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:05,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:05,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:05,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:05,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:05,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:05,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:05,806 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:05,806 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:05,806 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:05,806 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:05,806 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:05,808 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:05,808 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:05,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:05,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:05,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:05,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:05,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:05,808 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:05,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:05,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:05,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:05,809 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:05,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:05,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:05,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:05,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:05,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:05,810 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:05,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:05,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:05,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:05,812 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:05,812 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:05,812 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:05,812 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:05,812 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:05,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:05,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:05,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:05,812 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:05,812 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:05,812 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:05,812 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:05,812 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:05,812 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:05,812 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:05,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:05,813 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:05,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:05,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:05,913 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:05,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:06,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 1732612c92cd83e5f8169e1b3b27bd29 2023-04-09 09:54:06,708 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1732612c92cd83e5f8169e1b3b27bd29 3/3 column families, dataSize=140.89 KB heapSize=395.53 KB 2023-04-09 09:54:06,708 DEBUG [Thread-141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x02df095a to 127.0.0.1:57816 2023-04-09 09:54:06,708 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=A 2023-04-09 09:54:06,708 DEBUG [Thread-141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-09 09:54:06,708 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:54:06,708 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=B 2023-04-09 09:54:06,708 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:54:06,709 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=C 2023-04-09 09:54:06,709 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:54:06,719 DEBUG [Thread-143] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x61e72c30 to 127.0.0.1:57816 2023-04-09 09:54:06,719 DEBUG [Thread-143] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-09 09:54:07,117 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=761 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/A/fe56b322f40b490fb3ec040eb8aee1d5 2023-04-09 09:54:07,313 DEBUG [Thread-147] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4fea27db to 127.0.0.1:57816 2023-04-09 09:54:07,313 DEBUG [Thread-145] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0e16d41b to 127.0.0.1:57816 2023-04-09 09:54:07,313 DEBUG [Thread-147] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-09 09:54:07,313 DEBUG [Thread-145] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-09 09:54:07,333 DEBUG [Thread-149] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7e728705 to 127.0.0.1:57816 2023-04-09 09:54:07,333 DEBUG [Thread-149] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-09 09:54:07,333 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-09 09:54:07,334 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(394): wrote 35 2023-04-09 09:54:07,334 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(394): wrote 17 2023-04-09 09:54:07,334 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(394): wrote 179 2023-04-09 09:54:07,334 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(394): wrote 194 2023-04-09 09:54:07,334 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(394): wrote 222 2023-04-09 09:54:07,334 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-09 09:54:07,334 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(398): read 4537 2023-04-09 09:54:07,334 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(398): read 4428 2023-04-09 09:54:07,334 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-09 09:54:07,334 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(402): scanned 1962 2023-04-09 09:54:07,334 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(403): verified 5886 rows 2023-04-09 09:54:07,334 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(402): scanned 1966 2023-04-09 09:54:07,334 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(403): verified 5894 rows 2023-04-09 09:54:07,334 INFO [Listener at localhost/44563] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-09 09:54:07,334 DEBUG [Listener at localhost/44563] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6d61f291 to 127.0.0.1:57816 2023-04-09 09:54:07,334 DEBUG [Listener at localhost/44563] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-09 09:54:07,341 INFO [Listener at localhost/44563] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-09 09:54:07,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-09 09:54:07,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-09 09:54:07,356 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681034047356"}]},"ts":"1681034047356"} 2023-04-09 09:54:07,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-09 09:54:07,357 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-09 09:54:07,359 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-09 09:54:07,361 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=1732612c92cd83e5f8169e1b3b27bd29, UNASSIGN}] 2023-04-09 09:54:07,362 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=1732612c92cd83e5f8169e1b3b27bd29, UNASSIGN 2023-04-09 09:54:07,363 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=1732612c92cd83e5f8169e1b3b27bd29, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:07,363 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681034047363"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681034047363"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681034047363"}]},"ts":"1681034047363"} 2023-04-09 09:54:07,365 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure 1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544}] 2023-04-09 09:54:07,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-09 09:54:07,526 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 1732612c92cd83e5f8169e1b3b27bd29 2023-04-09 09:54:07,527 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1732612c92cd83e5f8169e1b3b27bd29, disabling compactions & flushes 2023-04-09 09:54:07,527 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:54:07,528 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=761 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/B/75bf92efea6942a2a3d6db6462ae9c40 2023-04-09 09:54:07,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-09 09:54:07,940 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=761 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/C/d1e379f0327a4c9193878b8e6459d50d 2023-04-09 09:54:07,945 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/A/fe56b322f40b490fb3ec040eb8aee1d5 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/fe56b322f40b490fb3ec040eb8aee1d5 2023-04-09 09:54:07,950 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/fe56b322f40b490fb3ec040eb8aee1d5, entries=150, sequenceid=761, filesize=11.9 K 2023-04-09 09:54:07,950 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/B/75bf92efea6942a2a3d6db6462ae9c40 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/75bf92efea6942a2a3d6db6462ae9c40 2023-04-09 09:54:07,955 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/75bf92efea6942a2a3d6db6462ae9c40, entries=150, sequenceid=761, filesize=11.9 K 2023-04-09 09:54:07,956 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/C/d1e379f0327a4c9193878b8e6459d50d as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/d1e379f0327a4c9193878b8e6459d50d 2023-04-09 09:54:07,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-09 09:54:07,960 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/d1e379f0327a4c9193878b8e6459d50d, entries=150, sequenceid=761, filesize=11.9 K 2023-04-09 09:54:07,960 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~395.48 KB/404976, currentSize=26.84 KB/27480 for 1732612c92cd83e5f8169e1b3b27bd29 in 1252ms, sequenceid=761, compaction requested=true 2023-04-09 09:54:07,961 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1732612c92cd83e5f8169e1b3b27bd29: 2023-04-09 09:54:07,961 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:54:07,961 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:54:07,961 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. because compaction request was cancelled 2023-04-09 09:54:07,961 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:54:07,961 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. because compaction request was cancelled 2023-04-09 09:54:07,961 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:54:07,961 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. after waiting 0 ms 2023-04-09 09:54:07,961 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:54:07,961 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:54:07,961 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. because compaction request was cancelled 2023-04-09 09:54:07,961 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1732612c92cd83e5f8169e1b3b27bd29 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-09 09:54:07,961 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=A 2023-04-09 09:54:07,961 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:07,961 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=B 2023-04-09 09:54:07,961 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:07,961 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1732612c92cd83e5f8169e1b3b27bd29, store=C 2023-04-09 09:54:07,961 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:08,368 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=768 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/A/aaff07216b854044b9f284c8665eb893 2023-04-09 09:54:08,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-09 09:54:08,780 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=768 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/B/fd2881a2d3b5444fb816f378bc128606 2023-04-09 09:54:09,191 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=768 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/C/79dd2cf85eab42268f8c8d278a993b03 2023-04-09 09:54:09,196 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/A/aaff07216b854044b9f284c8665eb893 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/aaff07216b854044b9f284c8665eb893 2023-04-09 09:54:09,202 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/aaff07216b854044b9f284c8665eb893, entries=100, sequenceid=768, filesize=9.5 K 2023-04-09 09:54:09,203 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/B/fd2881a2d3b5444fb816f378bc128606 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/fd2881a2d3b5444fb816f378bc128606 2023-04-09 09:54:09,207 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/fd2881a2d3b5444fb816f378bc128606, entries=100, sequenceid=768, filesize=9.5 K 2023-04-09 09:54:09,208 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/.tmp/C/79dd2cf85eab42268f8c8d278a993b03 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/79dd2cf85eab42268f8c8d278a993b03 2023-04-09 09:54:09,212 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/79dd2cf85eab42268f8c8d278a993b03, entries=100, sequenceid=768, filesize=9.5 K 2023-04-09 09:54:09,213 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 1732612c92cd83e5f8169e1b3b27bd29 in 1252ms, sequenceid=768, compaction requested=true 2023-04-09 09:54:09,234 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/2aa2e6bc555e493e8014f0b8d27cc471, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/a5cf92babdff45eb967f734cae0394be, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/a771e3e4779e41b09857a4ad421e0a46, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/f921476f471c463784c584e3a5ecbe41, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/7a9fa2fab9474a13813123784afb3d91, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/198606809714499ca6fca91e0ce38388, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/42572582c48b444ea0ae4f0fbc26d9c8, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/f1e1fd314ff84c439767c47794cc3b5c, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/eac6f83f3f9b415885e9347e0212bf22, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/e1bc9d9021fc40d3890429cf14316478, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/2b33909d70fd4427a0e4e0e231f7c2ea, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/039f4905f9c946cf9e2e2b12b56d1f53, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/6a62561777a64b4fbce46ee93ae54e21, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/01d62b04516b46998d8f1566f5d640cb, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/bb61906d5fa84fb58aade79bdeca85fa, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/75142d4b35234b77b8b69143120ae0cf, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/c70ec227af05431e8bc8bb661ba6c13a, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/2457b724b07a4d4baddda6fb98698332, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/2f7e0c607cc443908347c3ba6b5420bf, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/bd5328ad650a4d6cace5df36c0be431c, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/27e423631ae44e398a0385f92689488e, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/b2ea89d969a14aeeb324664a81c8a981, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/afadaee9accd445f81a0be9b4d8340a2, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/a602cab7afbe48cd9969f40505daad0b, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/d8a88c37a5164d61a7ccae323fe3ed2d, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/95c0fd3565da4dacbbf1be2d1b695e35, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/64225f79554c411b9bc55329d68423ab, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/987eccf7f23a4a1695a907015ed4cc96, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/734403ac7e26453b8c80e7a36b14254b, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/10b43aacc7a7459e91d4d037dd25810f, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/262ec3807c7446d497cae66a1b08f1e1, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/e91aea20cd8e41bd922966345330db5b, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/0893e2987ebf414c99553f57de4fa912] to archive 2023-04-09 09:54:09,235 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-09 09:54:09,241 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/2aa2e6bc555e493e8014f0b8d27cc471 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/2aa2e6bc555e493e8014f0b8d27cc471 2023-04-09 09:54:09,242 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/a5cf92babdff45eb967f734cae0394be to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/a5cf92babdff45eb967f734cae0394be 2023-04-09 09:54:09,243 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/a771e3e4779e41b09857a4ad421e0a46 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/a771e3e4779e41b09857a4ad421e0a46 2023-04-09 09:54:09,244 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/f921476f471c463784c584e3a5ecbe41 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/f921476f471c463784c584e3a5ecbe41 2023-04-09 09:54:09,246 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/7a9fa2fab9474a13813123784afb3d91 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/7a9fa2fab9474a13813123784afb3d91 2023-04-09 09:54:09,248 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/198606809714499ca6fca91e0ce38388 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/198606809714499ca6fca91e0ce38388 2023-04-09 09:54:09,249 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/42572582c48b444ea0ae4f0fbc26d9c8 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/42572582c48b444ea0ae4f0fbc26d9c8 2023-04-09 09:54:09,251 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/f1e1fd314ff84c439767c47794cc3b5c to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/f1e1fd314ff84c439767c47794cc3b5c 2023-04-09 09:54:09,252 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/eac6f83f3f9b415885e9347e0212bf22 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/eac6f83f3f9b415885e9347e0212bf22 2023-04-09 09:54:09,253 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/e1bc9d9021fc40d3890429cf14316478 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/e1bc9d9021fc40d3890429cf14316478 2023-04-09 09:54:09,254 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/2b33909d70fd4427a0e4e0e231f7c2ea to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/2b33909d70fd4427a0e4e0e231f7c2ea 2023-04-09 09:54:09,256 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/039f4905f9c946cf9e2e2b12b56d1f53 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/039f4905f9c946cf9e2e2b12b56d1f53 2023-04-09 09:54:09,257 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/6a62561777a64b4fbce46ee93ae54e21 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/6a62561777a64b4fbce46ee93ae54e21 2023-04-09 09:54:09,258 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/01d62b04516b46998d8f1566f5d640cb to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/01d62b04516b46998d8f1566f5d640cb 2023-04-09 09:54:09,259 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/bb61906d5fa84fb58aade79bdeca85fa to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/bb61906d5fa84fb58aade79bdeca85fa 2023-04-09 09:54:09,260 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/75142d4b35234b77b8b69143120ae0cf to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/75142d4b35234b77b8b69143120ae0cf 2023-04-09 09:54:09,261 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/c70ec227af05431e8bc8bb661ba6c13a to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/c70ec227af05431e8bc8bb661ba6c13a 2023-04-09 09:54:09,262 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/2457b724b07a4d4baddda6fb98698332 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/2457b724b07a4d4baddda6fb98698332 2023-04-09 09:54:09,264 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/2f7e0c607cc443908347c3ba6b5420bf to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/2f7e0c607cc443908347c3ba6b5420bf 2023-04-09 09:54:09,265 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/bd5328ad650a4d6cace5df36c0be431c to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/bd5328ad650a4d6cace5df36c0be431c 2023-04-09 09:54:09,266 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/27e423631ae44e398a0385f92689488e to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/27e423631ae44e398a0385f92689488e 2023-04-09 09:54:09,267 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/b2ea89d969a14aeeb324664a81c8a981 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/b2ea89d969a14aeeb324664a81c8a981 2023-04-09 09:54:09,268 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/afadaee9accd445f81a0be9b4d8340a2 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/afadaee9accd445f81a0be9b4d8340a2 2023-04-09 09:54:09,269 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/a602cab7afbe48cd9969f40505daad0b to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/a602cab7afbe48cd9969f40505daad0b 2023-04-09 09:54:09,270 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/d8a88c37a5164d61a7ccae323fe3ed2d to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/d8a88c37a5164d61a7ccae323fe3ed2d 2023-04-09 09:54:09,271 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/95c0fd3565da4dacbbf1be2d1b695e35 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/95c0fd3565da4dacbbf1be2d1b695e35 2023-04-09 09:54:09,272 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/64225f79554c411b9bc55329d68423ab to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/64225f79554c411b9bc55329d68423ab 2023-04-09 09:54:09,273 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/987eccf7f23a4a1695a907015ed4cc96 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/987eccf7f23a4a1695a907015ed4cc96 2023-04-09 09:54:09,274 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/734403ac7e26453b8c80e7a36b14254b to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/734403ac7e26453b8c80e7a36b14254b 2023-04-09 09:54:09,275 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/10b43aacc7a7459e91d4d037dd25810f to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/10b43aacc7a7459e91d4d037dd25810f 2023-04-09 09:54:09,276 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/262ec3807c7446d497cae66a1b08f1e1 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/262ec3807c7446d497cae66a1b08f1e1 2023-04-09 09:54:09,277 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/e91aea20cd8e41bd922966345330db5b to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/e91aea20cd8e41bd922966345330db5b 2023-04-09 09:54:09,278 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/0893e2987ebf414c99553f57de4fa912 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/0893e2987ebf414c99553f57de4fa912 2023-04-09 09:54:09,301 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/85f50da4b0584c62bdfca4d7eaeb0e51, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/1d59c66533ce4ef3b859497853e9980a, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/a5a9d5c7512846cfa9a15639f7777380, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/504f447e7bb54d5093883d200bb325f5, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/634319912aa64e459118c58d3da2bb56, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/6bba01064e294ec29be6740099bc6773, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/cf656a65e8454cc69f66df09b0316251, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/f02c5db68e234ff88bed1f1ba5cc318d, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/3d9d91e6b6a04224b63c91c3bcd0e2c1, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/cc558017a2384ea9b2cf5471581b116b, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/e4b770bdf6eb4b1387d3eec5bf6c253b, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/31d61467886642ca88348c46d1bc1902, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/bb99ae491f6b4deebd0913066acea7ef, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/9821d2ce7fd7400ebe1afc64dcb5d62a, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/ffd007ef7a0a480fa314b6b887e75d57, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/bc215c92b27f4bfcb2182f9fee5eeaa0, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/96a26d29b9e642a88f3156ed5953db8f, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/fd85c1ef748648379e8c0bbb223a2862, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/c000684a5bd5435da4125fafd3314f0d, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/d6a362a1e5084836a6db58515a58a467, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/793f7129fbc84e67a8d6eb9b658b2db3, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/cbd529705f4a4f0ab1d62cd9aa3c4c98, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/327f26834c044e32a9d079e451bb7db1, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/487f0ee29be8415c8775d4e605574f48, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/d5ce30c0e28b40f4aa1ad2b26d20cc2b, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/3582356e9b9a47eebefb2628826dc5a8, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/4cb965a085414c5ca8c8647e7ecf7c1a, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/a54c08b9597b429bbc533932d98a7702, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/e1274c3b707b49d38deadd6855f80477, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/9227135ca1f14ab5953d43331dc8a3fc, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/7e8678414ea64538b1c1d8c2f6e0e88b, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/3f63ff6b4ecd4df4bd660d2cf1cc4bb6, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/106807961ad643659d495644d89f1125] to archive 2023-04-09 09:54:09,302 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-09 09:54:09,303 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/85f50da4b0584c62bdfca4d7eaeb0e51 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/85f50da4b0584c62bdfca4d7eaeb0e51 2023-04-09 09:54:09,305 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/1d59c66533ce4ef3b859497853e9980a to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/1d59c66533ce4ef3b859497853e9980a 2023-04-09 09:54:09,306 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/a5a9d5c7512846cfa9a15639f7777380 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/a5a9d5c7512846cfa9a15639f7777380 2023-04-09 09:54:09,307 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/504f447e7bb54d5093883d200bb325f5 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/504f447e7bb54d5093883d200bb325f5 2023-04-09 09:54:09,308 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/634319912aa64e459118c58d3da2bb56 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/634319912aa64e459118c58d3da2bb56 2023-04-09 09:54:09,309 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/6bba01064e294ec29be6740099bc6773 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/6bba01064e294ec29be6740099bc6773 2023-04-09 09:54:09,310 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/cf656a65e8454cc69f66df09b0316251 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/cf656a65e8454cc69f66df09b0316251 2023-04-09 09:54:09,311 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/f02c5db68e234ff88bed1f1ba5cc318d to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/f02c5db68e234ff88bed1f1ba5cc318d 2023-04-09 09:54:09,312 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/3d9d91e6b6a04224b63c91c3bcd0e2c1 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/3d9d91e6b6a04224b63c91c3bcd0e2c1 2023-04-09 09:54:09,313 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/cc558017a2384ea9b2cf5471581b116b to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/cc558017a2384ea9b2cf5471581b116b 2023-04-09 09:54:09,314 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/e4b770bdf6eb4b1387d3eec5bf6c253b to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/e4b770bdf6eb4b1387d3eec5bf6c253b 2023-04-09 09:54:09,315 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/31d61467886642ca88348c46d1bc1902 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/31d61467886642ca88348c46d1bc1902 2023-04-09 09:54:09,316 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/bb99ae491f6b4deebd0913066acea7ef to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/bb99ae491f6b4deebd0913066acea7ef 2023-04-09 09:54:09,317 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/9821d2ce7fd7400ebe1afc64dcb5d62a to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/9821d2ce7fd7400ebe1afc64dcb5d62a 2023-04-09 09:54:09,318 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/ffd007ef7a0a480fa314b6b887e75d57 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/ffd007ef7a0a480fa314b6b887e75d57 2023-04-09 09:54:09,319 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/bc215c92b27f4bfcb2182f9fee5eeaa0 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/bc215c92b27f4bfcb2182f9fee5eeaa0 2023-04-09 09:54:09,319 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/96a26d29b9e642a88f3156ed5953db8f to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/96a26d29b9e642a88f3156ed5953db8f 2023-04-09 09:54:09,320 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/fd85c1ef748648379e8c0bbb223a2862 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/fd85c1ef748648379e8c0bbb223a2862 2023-04-09 09:54:09,321 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/c000684a5bd5435da4125fafd3314f0d to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/c000684a5bd5435da4125fafd3314f0d 2023-04-09 09:54:09,322 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/d6a362a1e5084836a6db58515a58a467 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/d6a362a1e5084836a6db58515a58a467 2023-04-09 09:54:09,323 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/793f7129fbc84e67a8d6eb9b658b2db3 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/793f7129fbc84e67a8d6eb9b658b2db3 2023-04-09 09:54:09,324 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/cbd529705f4a4f0ab1d62cd9aa3c4c98 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/cbd529705f4a4f0ab1d62cd9aa3c4c98 2023-04-09 09:54:09,325 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/327f26834c044e32a9d079e451bb7db1 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/327f26834c044e32a9d079e451bb7db1 2023-04-09 09:54:09,326 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/487f0ee29be8415c8775d4e605574f48 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/487f0ee29be8415c8775d4e605574f48 2023-04-09 09:54:09,327 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/d5ce30c0e28b40f4aa1ad2b26d20cc2b to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/d5ce30c0e28b40f4aa1ad2b26d20cc2b 2023-04-09 09:54:09,328 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/3582356e9b9a47eebefb2628826dc5a8 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/3582356e9b9a47eebefb2628826dc5a8 2023-04-09 09:54:09,329 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/4cb965a085414c5ca8c8647e7ecf7c1a to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/4cb965a085414c5ca8c8647e7ecf7c1a 2023-04-09 09:54:09,329 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/a54c08b9597b429bbc533932d98a7702 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/a54c08b9597b429bbc533932d98a7702 2023-04-09 09:54:09,330 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/e1274c3b707b49d38deadd6855f80477 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/e1274c3b707b49d38deadd6855f80477 2023-04-09 09:54:09,331 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/9227135ca1f14ab5953d43331dc8a3fc to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/9227135ca1f14ab5953d43331dc8a3fc 2023-04-09 09:54:09,332 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/7e8678414ea64538b1c1d8c2f6e0e88b to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/7e8678414ea64538b1c1d8c2f6e0e88b 2023-04-09 09:54:09,333 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/3f63ff6b4ecd4df4bd660d2cf1cc4bb6 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/3f63ff6b4ecd4df4bd660d2cf1cc4bb6 2023-04-09 09:54:09,334 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/106807961ad643659d495644d89f1125 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/106807961ad643659d495644d89f1125 2023-04-09 09:54:09,337 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/96ed90125b80487d81ff680cfd3a0fa9, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/869cddd98f354b2fa8a89dceaf19a738, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/13066ace425b40aebe907451acebb346, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/0b4aef4af45c4703becfe47daf81bdc9, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/1e1ff3bb1e5c4962afdaca6efa4de26a, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/d4962c47376c4769848588a5fbc9171c, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/822298757f664773aab2cf90d9e91f83, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/9cd565f14d5a47d0b9109c140dba6496, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/da1b99f29e1f4d5086a1fcb7f42ce6f0, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/ff5552e27d684f039711c65c622df4cc, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/6d643113098946c38e38f2ccc4c9ebd0, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/28c9b45329bc486ebdcdb42246ed1000, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/526fb8876b8f4b7e8b5041680767e823, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/6c7421348d3f4b69b77f41d770d1ad43, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/7a72d5983b58448fa0a3cca630c1275f, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/2b88ce5648544df1accedbf91c43d3b5, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/811704e8e43b41d3a22a3b5fe94ab0eb, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/06ac5987d7274e5db28161b21b47152a, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/53c65dcf65d041caa45cc6c4b0aecf52, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/5bb9109228f749d2b046a803fcf8e651, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/6aeaeb44f4c448148933064a9471b2cd, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/0e9b5ba1cfdc411096d1337f79fb345a, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/c282ada0bf7948f0a1813c115747f468, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/33407d10fa614cef843b4cab61bbddb0, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/a17924d94fa9457e89574fc641c9ffb4, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/c4a1ea2c4d614a5bad9db3d9bf95b240, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/e1daf8e3c0ff4680a12a25219bd2fac2, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/05dd1935f0174433a4c9bcf3fe0dac69, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/0a860ea9220a4f1ebbb8b0f27f40d966, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/a9089548e0704963bfc3e0491952b5f6, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/139c18deb6e94ed88bcc37c83ff0b1bc, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/c5cbd7daed4e475a8f61adb4cdbd21a0, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/ae4929ca81fc459080b9b6b3b9882e33] to archive 2023-04-09 09:54:09,338 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-09 09:54:09,339 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/96ed90125b80487d81ff680cfd3a0fa9 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/96ed90125b80487d81ff680cfd3a0fa9 2023-04-09 09:54:09,340 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/869cddd98f354b2fa8a89dceaf19a738 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/869cddd98f354b2fa8a89dceaf19a738 2023-04-09 09:54:09,341 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/13066ace425b40aebe907451acebb346 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/13066ace425b40aebe907451acebb346 2023-04-09 09:54:09,342 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/0b4aef4af45c4703becfe47daf81bdc9 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/0b4aef4af45c4703becfe47daf81bdc9 2023-04-09 09:54:09,343 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/1e1ff3bb1e5c4962afdaca6efa4de26a to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/1e1ff3bb1e5c4962afdaca6efa4de26a 2023-04-09 09:54:09,345 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/d4962c47376c4769848588a5fbc9171c to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/d4962c47376c4769848588a5fbc9171c 2023-04-09 09:54:09,346 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/822298757f664773aab2cf90d9e91f83 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/822298757f664773aab2cf90d9e91f83 2023-04-09 09:54:09,347 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/9cd565f14d5a47d0b9109c140dba6496 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/9cd565f14d5a47d0b9109c140dba6496 2023-04-09 09:54:09,347 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/da1b99f29e1f4d5086a1fcb7f42ce6f0 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/da1b99f29e1f4d5086a1fcb7f42ce6f0 2023-04-09 09:54:09,348 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/ff5552e27d684f039711c65c622df4cc to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/ff5552e27d684f039711c65c622df4cc 2023-04-09 09:54:09,349 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/6d643113098946c38e38f2ccc4c9ebd0 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/6d643113098946c38e38f2ccc4c9ebd0 2023-04-09 09:54:09,350 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/28c9b45329bc486ebdcdb42246ed1000 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/28c9b45329bc486ebdcdb42246ed1000 2023-04-09 09:54:09,351 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/526fb8876b8f4b7e8b5041680767e823 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/526fb8876b8f4b7e8b5041680767e823 2023-04-09 09:54:09,352 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/6c7421348d3f4b69b77f41d770d1ad43 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/6c7421348d3f4b69b77f41d770d1ad43 2023-04-09 09:54:09,353 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/7a72d5983b58448fa0a3cca630c1275f to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/7a72d5983b58448fa0a3cca630c1275f 2023-04-09 09:54:09,354 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/2b88ce5648544df1accedbf91c43d3b5 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/2b88ce5648544df1accedbf91c43d3b5 2023-04-09 09:54:09,355 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/811704e8e43b41d3a22a3b5fe94ab0eb to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/811704e8e43b41d3a22a3b5fe94ab0eb 2023-04-09 09:54:09,356 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/06ac5987d7274e5db28161b21b47152a to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/06ac5987d7274e5db28161b21b47152a 2023-04-09 09:54:09,356 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/53c65dcf65d041caa45cc6c4b0aecf52 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/53c65dcf65d041caa45cc6c4b0aecf52 2023-04-09 09:54:09,357 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/5bb9109228f749d2b046a803fcf8e651 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/5bb9109228f749d2b046a803fcf8e651 2023-04-09 09:54:09,358 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/6aeaeb44f4c448148933064a9471b2cd to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/6aeaeb44f4c448148933064a9471b2cd 2023-04-09 09:54:09,360 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/0e9b5ba1cfdc411096d1337f79fb345a to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/0e9b5ba1cfdc411096d1337f79fb345a 2023-04-09 09:54:09,361 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/c282ada0bf7948f0a1813c115747f468 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/c282ada0bf7948f0a1813c115747f468 2023-04-09 09:54:09,362 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/33407d10fa614cef843b4cab61bbddb0 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/33407d10fa614cef843b4cab61bbddb0 2023-04-09 09:54:09,363 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/a17924d94fa9457e89574fc641c9ffb4 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/a17924d94fa9457e89574fc641c9ffb4 2023-04-09 09:54:09,364 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/c4a1ea2c4d614a5bad9db3d9bf95b240 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/c4a1ea2c4d614a5bad9db3d9bf95b240 2023-04-09 09:54:09,365 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/e1daf8e3c0ff4680a12a25219bd2fac2 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/e1daf8e3c0ff4680a12a25219bd2fac2 2023-04-09 09:54:09,366 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/05dd1935f0174433a4c9bcf3fe0dac69 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/05dd1935f0174433a4c9bcf3fe0dac69 2023-04-09 09:54:09,367 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/0a860ea9220a4f1ebbb8b0f27f40d966 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/0a860ea9220a4f1ebbb8b0f27f40d966 2023-04-09 09:54:09,369 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/a9089548e0704963bfc3e0491952b5f6 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/a9089548e0704963bfc3e0491952b5f6 2023-04-09 09:54:09,370 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/139c18deb6e94ed88bcc37c83ff0b1bc to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/139c18deb6e94ed88bcc37c83ff0b1bc 2023-04-09 09:54:09,371 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/c5cbd7daed4e475a8f61adb4cdbd21a0 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/c5cbd7daed4e475a8f61adb4cdbd21a0 2023-04-09 09:54:09,373 DEBUG [StoreCloser-TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/ae4929ca81fc459080b9b6b3b9882e33 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/ae4929ca81fc459080b9b6b3b9882e33 2023-04-09 09:54:09,378 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/recovered.edits/771.seqid, newMaxSeqId=771, maxSeqId=1 2023-04-09 09:54:09,379 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29. 2023-04-09 09:54:09,379 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1732612c92cd83e5f8169e1b3b27bd29: 2023-04-09 09:54:09,381 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 1732612c92cd83e5f8169e1b3b27bd29 2023-04-09 09:54:09,381 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=1732612c92cd83e5f8169e1b3b27bd29, regionState=CLOSED 2023-04-09 09:54:09,382 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681034049381"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681034049381"}]},"ts":"1681034049381"} 2023-04-09 09:54:09,384 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-04-09 09:54:09,385 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure 1732612c92cd83e5f8169e1b3b27bd29, server=jenkins-hbase4.apache.org,42467,1681034019544 in 2.0180 sec 2023-04-09 09:54:09,386 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-04-09 09:54:09,386 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=1732612c92cd83e5f8169e1b3b27bd29, UNASSIGN in 2.0240 sec 2023-04-09 09:54:09,387 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681034049387"}]},"ts":"1681034049387"} 2023-04-09 09:54:09,388 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-09 09:54:09,392 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-09 09:54:09,394 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.0440 sec 2023-04-09 09:54:09,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-09 09:54:09,462 INFO [Listener at localhost/44563] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 12 completed 2023-04-09 09:54:09,466 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-09 09:54:09,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-09 09:54:09,474 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-09 09:54:09,476 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-09 09:54:09,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-09 09:54:09,480 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29 2023-04-09 09:54:09,483 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A, FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B, FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C, FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/recovered.edits] 2023-04-09 09:54:09,487 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/05ff574f39424b92b0babf4c85a6e574 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/05ff574f39424b92b0babf4c85a6e574 2023-04-09 09:54:09,488 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/14ef4962bdb1439ea4d02653ab3b65f1 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/14ef4962bdb1439ea4d02653ab3b65f1 2023-04-09 09:54:09,489 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/842c7182675a41bd992dd2c868722d00 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/842c7182675a41bd992dd2c868722d00 2023-04-09 09:54:09,490 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/aaff07216b854044b9f284c8665eb893 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/aaff07216b854044b9f284c8665eb893 2023-04-09 09:54:09,492 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/fe56b322f40b490fb3ec040eb8aee1d5 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/A/fe56b322f40b490fb3ec040eb8aee1d5 2023-04-09 09:54:09,494 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/14582e1241b54e5283fc3d084ea4d525 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/14582e1241b54e5283fc3d084ea4d525 2023-04-09 09:54:09,495 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/4ab0f8ad0a774c65bd2427f47c55b986 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/4ab0f8ad0a774c65bd2427f47c55b986 2023-04-09 09:54:09,497 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/6957a78ab87a431fad086a7395f5eb94 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/6957a78ab87a431fad086a7395f5eb94 2023-04-09 09:54:09,498 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/75bf92efea6942a2a3d6db6462ae9c40 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/75bf92efea6942a2a3d6db6462ae9c40 2023-04-09 09:54:09,499 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/fd2881a2d3b5444fb816f378bc128606 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/B/fd2881a2d3b5444fb816f378bc128606 2023-04-09 09:54:09,502 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/196a5464b72548daa651189948fdd7f4 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/196a5464b72548daa651189948fdd7f4 2023-04-09 09:54:09,503 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/1e0a5a5c96a14601b47e0f63addedf1d to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/1e0a5a5c96a14601b47e0f63addedf1d 2023-04-09 09:54:09,505 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/5c3836131ca7494e9dec0068135ed226 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/5c3836131ca7494e9dec0068135ed226 2023-04-09 09:54:09,506 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/79dd2cf85eab42268f8c8d278a993b03 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/79dd2cf85eab42268f8c8d278a993b03 2023-04-09 09:54:09,507 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/d1e379f0327a4c9193878b8e6459d50d to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/C/d1e379f0327a4c9193878b8e6459d50d 2023-04-09 09:54:09,510 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/recovered.edits/771.seqid to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29/recovered.edits/771.seqid 2023-04-09 09:54:09,511 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/1732612c92cd83e5f8169e1b3b27bd29 2023-04-09 09:54:09,511 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-09 09:54:09,513 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-09 09:54:09,534 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-09 09:54:09,576 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-09 09:54:09,578 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-09 09:54:09,578 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-09 09:54:09,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-09 09:54:09,578 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681034049578"}]},"ts":"9223372036854775807"} 2023-04-09 09:54:09,580 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-09 09:54:09,580 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 1732612c92cd83e5f8169e1b3b27bd29, NAME => 'TestAcidGuarantees,,1681034023887.1732612c92cd83e5f8169e1b3b27bd29.', STARTKEY => '', ENDKEY => ''}] 2023-04-09 09:54:09,580 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-09 09:54:09,580 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681034049580"}]},"ts":"9223372036854775807"} 2023-04-09 09:54:09,582 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-09 09:54:09,584 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-09 09:54:09,585 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 116 msec 2023-04-09 09:54:09,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-09 09:54:09,779 INFO [Listener at localhost/44563] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 15 completed 2023-04-09 09:54:09,796 INFO [Listener at localhost/44563] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=236 (was 207) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x12cc5459-shared-pool-20 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_618788791_15 at /127.0.0.1:49430 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: (jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x12cc5459-shared-pool-21 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_618788791_15 at /127.0.0.1:49478 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_618788791_15 at /127.0.0.1:49524 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x12cc5459-shared-pool-18 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x12cc5459-shared-pool-19 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:0;jenkins-hbase4:42467-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=561 (was 470) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=426 (was 301) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=1883 (was 2520) 2023-04-09 09:54:09,806 INFO [Listener at localhost/44563] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=236, OpenFileDescriptor=561, MaxFileDescriptor=60000, SystemLoadAverage=426, ProcessCount=169, AvailableMemoryMB=1882 2023-04-09 09:54:09,808 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-09 09:54:09,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-09 09:54:09,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure2.ProcedureExecutor(1029): Stored pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-09 09:54:09,811 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-09 09:54:09,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 16 2023-04-09 09:54:09,812 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-09 09:54:09,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-09 09:54:09,814 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40 2023-04-09 09:54:09,814 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40 empty. 2023-04-09 09:54:09,815 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40 2023-04-09 09:54:09,815 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-09 09:54:09,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-09 09:54:10,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-09 09:54:10,225 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-09 09:54:10,227 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => b86d94f6f111f1359f84891af8d12a40, NAME => 'TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp 2023-04-09 09:54:10,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-09 09:54:10,633 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-09 09:54:10,633 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing b86d94f6f111f1359f84891af8d12a40, disabling compactions & flushes 2023-04-09 09:54:10,633 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:10,633 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:10,633 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. after waiting 0 ms 2023-04-09 09:54:10,634 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:10,634 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:10,634 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for b86d94f6f111f1359f84891af8d12a40: 2023-04-09 09:54:10,636 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-09 09:54:10,637 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681034050637"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681034050637"}]},"ts":"1681034050637"} 2023-04-09 09:54:10,638 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-09 09:54:10,639 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-09 09:54:10,639 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681034050639"}]},"ts":"1681034050639"} 2023-04-09 09:54:10,640 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-09 09:54:10,644 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=b86d94f6f111f1359f84891af8d12a40, ASSIGN}] 2023-04-09 09:54:10,646 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=b86d94f6f111f1359f84891af8d12a40, ASSIGN 2023-04-09 09:54:10,646 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=b86d94f6f111f1359f84891af8d12a40, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,42467,1681034019544; forceNewPlan=false, retain=false 2023-04-09 09:54:10,797 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=b86d94f6f111f1359f84891af8d12a40, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:10,797 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681034050797"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681034050797"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681034050797"}]},"ts":"1681034050797"} 2023-04-09 09:54:10,799 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; OpenRegionProcedure b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544}] 2023-04-09 09:54:10,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-09 09:54:10,956 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:10,957 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => b86d94f6f111f1359f84891af8d12a40, NAME => 'TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.', STARTKEY => '', ENDKEY => ''} 2023-04-09 09:54:10,957 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees b86d94f6f111f1359f84891af8d12a40 2023-04-09 09:54:10,957 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-09 09:54:10,957 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for b86d94f6f111f1359f84891af8d12a40 2023-04-09 09:54:10,957 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for b86d94f6f111f1359f84891af8d12a40 2023-04-09 09:54:10,959 INFO [StoreOpener-b86d94f6f111f1359f84891af8d12a40-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region b86d94f6f111f1359f84891af8d12a40 2023-04-09 09:54:10,961 DEBUG [StoreOpener-b86d94f6f111f1359f84891af8d12a40-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A 2023-04-09 09:54:10,961 DEBUG [StoreOpener-b86d94f6f111f1359f84891af8d12a40-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A 2023-04-09 09:54:10,961 INFO [StoreOpener-b86d94f6f111f1359f84891af8d12a40-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-09 09:54:10,962 INFO [StoreOpener-b86d94f6f111f1359f84891af8d12a40-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b86d94f6f111f1359f84891af8d12a40 columnFamilyName A 2023-04-09 09:54:10,963 INFO [StoreOpener-b86d94f6f111f1359f84891af8d12a40-1] regionserver.HStore(310): Store=b86d94f6f111f1359f84891af8d12a40/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-09 09:54:10,963 INFO [StoreOpener-b86d94f6f111f1359f84891af8d12a40-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region b86d94f6f111f1359f84891af8d12a40 2023-04-09 09:54:10,965 DEBUG [StoreOpener-b86d94f6f111f1359f84891af8d12a40-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B 2023-04-09 09:54:10,965 DEBUG [StoreOpener-b86d94f6f111f1359f84891af8d12a40-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B 2023-04-09 09:54:10,965 INFO [StoreOpener-b86d94f6f111f1359f84891af8d12a40-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-09 09:54:10,965 INFO [StoreOpener-b86d94f6f111f1359f84891af8d12a40-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b86d94f6f111f1359f84891af8d12a40 columnFamilyName B 2023-04-09 09:54:10,966 INFO [StoreOpener-b86d94f6f111f1359f84891af8d12a40-1] regionserver.HStore(310): Store=b86d94f6f111f1359f84891af8d12a40/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-09 09:54:10,966 INFO [StoreOpener-b86d94f6f111f1359f84891af8d12a40-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region b86d94f6f111f1359f84891af8d12a40 2023-04-09 09:54:10,967 DEBUG [StoreOpener-b86d94f6f111f1359f84891af8d12a40-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C 2023-04-09 09:54:10,967 DEBUG [StoreOpener-b86d94f6f111f1359f84891af8d12a40-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C 2023-04-09 09:54:10,967 INFO [StoreOpener-b86d94f6f111f1359f84891af8d12a40-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-09 09:54:10,968 INFO [StoreOpener-b86d94f6f111f1359f84891af8d12a40-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b86d94f6f111f1359f84891af8d12a40 columnFamilyName C 2023-04-09 09:54:10,968 INFO [StoreOpener-b86d94f6f111f1359f84891af8d12a40-1] regionserver.HStore(310): Store=b86d94f6f111f1359f84891af8d12a40/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-09 09:54:10,968 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:10,969 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40 2023-04-09 09:54:10,970 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40 2023-04-09 09:54:10,972 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-09 09:54:10,973 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for b86d94f6f111f1359f84891af8d12a40 2023-04-09 09:54:10,975 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-09 09:54:10,976 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened b86d94f6f111f1359f84891af8d12a40; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68189167, jitterRate=0.016097769141197205}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-09 09:54:10,976 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for b86d94f6f111f1359f84891af8d12a40: 2023-04-09 09:54:10,977 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40., pid=18, masterSystemTime=1681034050951 2023-04-09 09:54:10,978 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:10,979 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:10,979 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=b86d94f6f111f1359f84891af8d12a40, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:10,979 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681034050979"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681034050979"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681034050979"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681034050979"}]},"ts":"1681034050979"} 2023-04-09 09:54:10,982 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-04-09 09:54:10,982 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; OpenRegionProcedure b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 in 182 msec 2023-04-09 09:54:10,984 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-04-09 09:54:10,984 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=b86d94f6f111f1359f84891af8d12a40, ASSIGN in 338 msec 2023-04-09 09:54:10,984 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-09 09:54:10,985 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681034050985"}]},"ts":"1681034050985"} 2023-04-09 09:54:10,986 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-09 09:54:10,989 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-09 09:54:10,990 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=16, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1800 sec 2023-04-09 09:54:11,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-09 09:54:11,918 INFO [Listener at localhost/44563] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 16 completed 2023-04-09 09:54:11,919 DEBUG [Listener at localhost/44563] zookeeper.ReadOnlyZKClient(139): Connect 0x31436cf6 to 127.0.0.1:57816 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-09 09:54:11,923 DEBUG [Listener at localhost/44563] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@33c943f0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-09 09:54:11,925 DEBUG [Listener at localhost/44563] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-09 09:54:11,928 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59740, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-09 09:54:11,930 DEBUG [Listener at localhost/44563] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-09 09:54:11,932 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59742, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-09 09:54:11,941 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-09 09:54:11,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-09 09:54:11,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-09 09:54:12,363 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-09 09:54:12,364 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-09 09:54:12,364 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-09 09:54:12,368 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-09 09:54:12,381 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=b86d94f6f111f1359f84891af8d12a40, REOPEN/MOVE}] 2023-04-09 09:54:12,382 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=b86d94f6f111f1359f84891af8d12a40, REOPEN/MOVE 2023-04-09 09:54:12,383 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=b86d94f6f111f1359f84891af8d12a40, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:12,383 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681034052383"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681034052383"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681034052383"}]},"ts":"1681034052383"} 2023-04-09 09:54:12,384 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; CloseRegionProcedure b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544}] 2023-04-09 09:54:12,536 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close b86d94f6f111f1359f84891af8d12a40 2023-04-09 09:54:12,537 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing b86d94f6f111f1359f84891af8d12a40, disabling compactions & flushes 2023-04-09 09:54:12,537 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:12,537 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:12,537 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. after waiting 0 ms 2023-04-09 09:54:12,537 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:12,541 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-09 09:54:12,541 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:12,542 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for b86d94f6f111f1359f84891af8d12a40: 2023-04-09 09:54:12,542 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: b86d94f6f111f1359f84891af8d12a40 to self. 2023-04-09 09:54:12,543 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed b86d94f6f111f1359f84891af8d12a40 2023-04-09 09:54:12,544 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=b86d94f6f111f1359f84891af8d12a40, regionState=CLOSED 2023-04-09 09:54:12,544 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681034052543"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681034052543"}]},"ts":"1681034052543"} 2023-04-09 09:54:12,546 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-04-09 09:54:12,546 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; CloseRegionProcedure b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 in 161 msec 2023-04-09 09:54:12,547 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=b86d94f6f111f1359f84891af8d12a40, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,42467,1681034019544; forceNewPlan=false, retain=true 2023-04-09 09:54:12,697 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=b86d94f6f111f1359f84891af8d12a40, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:12,697 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681034052697"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681034052697"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681034052697"}]},"ts":"1681034052697"} 2023-04-09 09:54:12,699 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=21, state=RUNNABLE; OpenRegionProcedure b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544}] 2023-04-09 09:54:12,854 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:12,854 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => b86d94f6f111f1359f84891af8d12a40, NAME => 'TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.', STARTKEY => '', ENDKEY => ''} 2023-04-09 09:54:12,855 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees b86d94f6f111f1359f84891af8d12a40 2023-04-09 09:54:12,855 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-09 09:54:12,855 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for b86d94f6f111f1359f84891af8d12a40 2023-04-09 09:54:12,855 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for b86d94f6f111f1359f84891af8d12a40 2023-04-09 09:54:12,858 INFO [StoreOpener-b86d94f6f111f1359f84891af8d12a40-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region b86d94f6f111f1359f84891af8d12a40 2023-04-09 09:54:12,858 DEBUG [StoreOpener-b86d94f6f111f1359f84891af8d12a40-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A 2023-04-09 09:54:12,858 DEBUG [StoreOpener-b86d94f6f111f1359f84891af8d12a40-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A 2023-04-09 09:54:12,859 INFO [StoreOpener-b86d94f6f111f1359f84891af8d12a40-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-09 09:54:12,861 INFO [StoreOpener-b86d94f6f111f1359f84891af8d12a40-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b86d94f6f111f1359f84891af8d12a40 columnFamilyName A 2023-04-09 09:54:12,863 INFO [StoreOpener-b86d94f6f111f1359f84891af8d12a40-1] regionserver.HStore(310): Store=b86d94f6f111f1359f84891af8d12a40/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-09 09:54:12,864 INFO [StoreOpener-b86d94f6f111f1359f84891af8d12a40-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region b86d94f6f111f1359f84891af8d12a40 2023-04-09 09:54:12,864 DEBUG [StoreOpener-b86d94f6f111f1359f84891af8d12a40-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B 2023-04-09 09:54:12,865 DEBUG [StoreOpener-b86d94f6f111f1359f84891af8d12a40-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B 2023-04-09 09:54:12,865 INFO [StoreOpener-b86d94f6f111f1359f84891af8d12a40-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-09 09:54:12,865 INFO [StoreOpener-b86d94f6f111f1359f84891af8d12a40-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b86d94f6f111f1359f84891af8d12a40 columnFamilyName B 2023-04-09 09:54:12,866 INFO [StoreOpener-b86d94f6f111f1359f84891af8d12a40-1] regionserver.HStore(310): Store=b86d94f6f111f1359f84891af8d12a40/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-09 09:54:12,866 INFO [StoreOpener-b86d94f6f111f1359f84891af8d12a40-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region b86d94f6f111f1359f84891af8d12a40 2023-04-09 09:54:12,866 DEBUG [StoreOpener-b86d94f6f111f1359f84891af8d12a40-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C 2023-04-09 09:54:12,866 DEBUG [StoreOpener-b86d94f6f111f1359f84891af8d12a40-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C 2023-04-09 09:54:12,866 INFO [StoreOpener-b86d94f6f111f1359f84891af8d12a40-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-09 09:54:12,867 INFO [StoreOpener-b86d94f6f111f1359f84891af8d12a40-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b86d94f6f111f1359f84891af8d12a40 columnFamilyName C 2023-04-09 09:54:12,868 INFO [StoreOpener-b86d94f6f111f1359f84891af8d12a40-1] regionserver.HStore(310): Store=b86d94f6f111f1359f84891af8d12a40/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-09 09:54:12,868 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:12,868 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40 2023-04-09 09:54:12,869 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40 2023-04-09 09:54:12,871 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-09 09:54:12,872 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for b86d94f6f111f1359f84891af8d12a40 2023-04-09 09:54:12,873 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened b86d94f6f111f1359f84891af8d12a40; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64947273, jitterRate=-0.03221021592617035}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-09 09:54:12,873 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for b86d94f6f111f1359f84891af8d12a40: 2023-04-09 09:54:12,873 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40., pid=23, masterSystemTime=1681034052851 2023-04-09 09:54:12,875 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:12,875 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:12,875 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=b86d94f6f111f1359f84891af8d12a40, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:12,876 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681034052875"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681034052875"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681034052875"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681034052875"}]},"ts":"1681034052875"} 2023-04-09 09:54:12,878 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=21 2023-04-09 09:54:12,878 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=21, state=SUCCESS; OpenRegionProcedure b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 in 178 msec 2023-04-09 09:54:12,880 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-04-09 09:54:12,880 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=b86d94f6f111f1359f84891af8d12a40, REOPEN/MOVE in 497 msec 2023-04-09 09:54:12,882 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-04-09 09:54:12,882 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 514 msec 2023-04-09 09:54:12,884 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 939 msec 2023-04-09 09:54:12,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-04-09 09:54:12,894 DEBUG [Listener at localhost/44563] zookeeper.ReadOnlyZKClient(139): Connect 0x29f78b2c to 127.0.0.1:57816 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-09 09:54:12,899 DEBUG [Listener at localhost/44563] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@51c62b6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-09 09:54:12,900 DEBUG [Listener at localhost/44563] zookeeper.ReadOnlyZKClient(139): Connect 0x7bbdba17 to 127.0.0.1:57816 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-09 09:54:12,906 DEBUG [Listener at localhost/44563] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1aa5eef5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-09 09:54:12,906 DEBUG [Listener at localhost/44563] zookeeper.ReadOnlyZKClient(139): Connect 0x02bab6b2 to 127.0.0.1:57816 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-09 09:54:12,909 DEBUG [Listener at localhost/44563] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1441c9f0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-09 09:54:12,910 DEBUG [Listener at localhost/44563] zookeeper.ReadOnlyZKClient(139): Connect 0x73e51571 to 127.0.0.1:57816 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-09 09:54:12,914 DEBUG [Listener at localhost/44563] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@69f15cb4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-09 09:54:12,914 DEBUG [Listener at localhost/44563] zookeeper.ReadOnlyZKClient(139): Connect 0x370947cb to 127.0.0.1:57816 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-09 09:54:12,917 DEBUG [Listener at localhost/44563] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6c1c92c6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-09 09:54:12,918 DEBUG [Listener at localhost/44563] zookeeper.ReadOnlyZKClient(139): Connect 0x5c472cfd to 127.0.0.1:57816 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-09 09:54:12,923 DEBUG [Listener at localhost/44563] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@54451f11, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-09 09:54:12,923 DEBUG [Listener at localhost/44563] zookeeper.ReadOnlyZKClient(139): Connect 0x498694cb to 127.0.0.1:57816 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-09 09:54:12,926 DEBUG [Listener at localhost/44563] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2c4379c2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-09 09:54:12,926 DEBUG [Listener at localhost/44563] zookeeper.ReadOnlyZKClient(139): Connect 0x421e5356 to 127.0.0.1:57816 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-09 09:54:12,929 DEBUG [Listener at localhost/44563] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3f43b712, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-09 09:54:12,929 DEBUG [Listener at localhost/44563] zookeeper.ReadOnlyZKClient(139): Connect 0x4a0fd7b4 to 127.0.0.1:57816 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-09 09:54:12,933 DEBUG [Listener at localhost/44563] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7a3bb940, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-09 09:54:12,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:12,941 DEBUG [hconnection-0x3a231f0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-09 09:54:12,942 DEBUG [hconnection-0x535b67f1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-09 09:54:12,942 DEBUG [hconnection-0x21ee8d8b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-09 09:54:12,944 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59752, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-09 09:54:12,945 DEBUG [hconnection-0x7fe5ac31-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-09 09:54:12,946 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59786, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-09 09:54:12,947 DEBUG [hconnection-0x640b41de-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-09 09:54:12,947 DEBUG [hconnection-0x621b88cf-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-09 09:54:12,950 DEBUG [hconnection-0x50f51887-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-09 09:54:12,950 DEBUG [hconnection-0x73ba6b50-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-09 09:54:12,951 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59816, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-09 09:54:12,954 DEBUG [hconnection-0x69d4ad36-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-09 09:54:12,955 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59770, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-09 09:54:12,955 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59766, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-09 09:54:12,955 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59804, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-09 09:54:12,955 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59788, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-09 09:54:12,955 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59820, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-09 09:54:12,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:12,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:12,962 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:12,962 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:12,962 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:12,962 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:12,962 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59822, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-09 09:54:12,966 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:12,966 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:12,969 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:12,969 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:12,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:12,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:12,970 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:12,970 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:12,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:12,971 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:12,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:12,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:12,971 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:12,974 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:12,974 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:12,974 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:12,974 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:12,974 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:12,976 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:12,976 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:12,976 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:12,976 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. started... 2023-04-09 09:54:12,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on b86d94f6f111f1359f84891af8d12a40 2023-04-09 09:54:12,977 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. as already flushing 2023-04-09 09:54:12,977 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b86d94f6f111f1359f84891af8d12a40 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-09 09:54:12,984 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=A 2023-04-09 09:54:12,984 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:12,984 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=B 2023-04-09 09:54:12,984 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:12,985 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=C 2023-04-09 09:54:12,985 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:13,006 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:13,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59752 deadline: 1681034113001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,007 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:13,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59822 deadline: 1681034113002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,007 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:13,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59766 deadline: 1681034113003, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,008 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:13,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59788 deadline: 1681034113003, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,008 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:13,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59804 deadline: 1681034113004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,062 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304090047777d581a43838e3d54bcbce7438a to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304090047777d581a43838e3d54bcbce7438a 2023-04-09 09:54:13,092 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=16, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/A/197873c5abc04c129b4e40349c8944c5 2023-04-09 09:54:13,109 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:13,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59822 deadline: 1681034113108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,109 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:13,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59766 deadline: 1681034113109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,109 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:13,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59752 deadline: 1681034113109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,110 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:13,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59788 deadline: 1681034113109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,110 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:13,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59804 deadline: 1681034113110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,156 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/B/4f639fd99d8f423c8021f57d15986701 2023-04-09 09:54:13,182 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/C/212195da92fc4eaeadc2aa3ef754b12a 2023-04-09 09:54:13,189 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/A/197873c5abc04c129b4e40349c8944c5 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/197873c5abc04c129b4e40349c8944c5 2023-04-09 09:54:13,195 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/197873c5abc04c129b4e40349c8944c5, entries=150, sequenceid=16, filesize=25.1 K 2023-04-09 09:54:13,196 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/B/4f639fd99d8f423c8021f57d15986701 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/4f639fd99d8f423c8021f57d15986701 2023-04-09 09:54:13,202 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/4f639fd99d8f423c8021f57d15986701, entries=150, sequenceid=16, filesize=11.6 K 2023-04-09 09:54:13,204 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/C/212195da92fc4eaeadc2aa3ef754b12a as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/212195da92fc4eaeadc2aa3ef754b12a 2023-04-09 09:54:13,209 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/212195da92fc4eaeadc2aa3ef754b12a, entries=150, sequenceid=16, filesize=11.6 K 2023-04-09 09:54:13,210 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for b86d94f6f111f1359f84891af8d12a40 in 233ms, sequenceid=16, compaction requested=false 2023-04-09 09:54:13,210 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b86d94f6f111f1359f84891af8d12a40: 2023-04-09 09:54:13,210 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:13,210 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:13,210 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:13,210 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,212 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:13,212 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:13,213 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,213 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:13,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:13,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:13,214 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:13,214 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:13,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:13,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:13,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:13,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:13,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:13,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@249c4765[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:13,215 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:13,215 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:13,217 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:13,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:13,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:13,217 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:13,217 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:13,217 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,217 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:13,217 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:13,218 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:13,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:13,219 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:13,219 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:13,219 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:13,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:13,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:13,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:13,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:13,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:13,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:13,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:13,222 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:13,222 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:13,222 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:13,222 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:13,222 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:13,225 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:13,225 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:13,225 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:13,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:13,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:13,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:13,226 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:13,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:13,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:13,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:13,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:13,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:13,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:13,227 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:13,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:13,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:13,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:13,230 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:13,230 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:13,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:13,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:13,231 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,231 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:13,231 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:13,231 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,231 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:13,231 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:13,231 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:13,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:13,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:13,231 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:13,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:13,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:13,231 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:13,231 DEBUG [Thread-888] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:13,231 DEBUG [Thread-888] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:13,314 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034053001, max timestamp=1681034053311Num uniques 859; 2023-04-09 09:54:13,315 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:54:13,315 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034053001, max timestamp=1681034053313Num uniques 859; 2023-04-09 09:54:13,315 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:54:13,316 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b86d94f6f111f1359f84891af8d12a40 3/3 column families, dataSize=58.14 KB heapSize=164.17 KB 2023-04-09 09:54:13,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on b86d94f6f111f1359f84891af8d12a40 2023-04-09 09:54:13,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034053001, max timestamp=1681034053313Num uniques 859; 2023-04-09 09:54:13,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:54:13,317 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=A 2023-04-09 09:54:13,317 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:54:13,317 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=B 2023-04-09 09:54:13,321 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:54:13,321 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=C 2023-04-09 09:54:13,321 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:54:13,332 DEBUG [Thread-888] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:13,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:13,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:13,348 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681034053337, max timestamp=1681034053345Num uniques 859; 2023-04-09 09:54:13,348 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-09 09:54:13,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681034053337, max timestamp=1681034053344Num uniques 859; 2023-04-09 09:54:13,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034053337, max timestamp=1681034053344Num uniques 859; 2023-04-09 09:54:13,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-09 09:54:13,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:54:13,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:13,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:13,368 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:13,368 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:13,368 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:13,368 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:13,369 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:13,369 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:13,370 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,370 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:13,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:13,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:13,370 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,370 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:13,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:13,371 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:13,374 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1681034053365, max timestamp=1681034053370Num uniques 1068; 2023-04-09 09:54:13,374 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=14558, suffix heap size=139040,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=318 2023-04-09 09:54:13,374 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1681034053365, max timestamp=1681034053370Num uniques 1018; 2023-04-09 09:54:13,374 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=14558, suffix heap size=133840,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=318 2023-04-09 09:54:13,374 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1681034053365, max timestamp=1681034053370Num uniques 1068; 2023-04-09 09:54:13,375 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=14558, suffix heap size=139040,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=318 2023-04-09 09:54:13,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:13,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:13,375 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:13,384 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:13,384 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:13,385 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:13,385 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:13,385 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:13,392 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:13,392 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:13,392 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:13,392 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. started... 2023-04-09 09:54:13,392 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. as already flushing 2023-04-09 09:54:13,392 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:13,392 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:13,392 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:13,392 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,397 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:13,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59752 deadline: 1681034113394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,397 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681034053392, max timestamp=1681034053394Num uniques 1177; 2023-04-09 09:54:13,397 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:13,397 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=53902, new segment data size=8102, suffix heap size=150376,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1177, new segment cells count=177 2023-04-09 09:54:13,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59804 deadline: 1681034113394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,398 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681034053392, max timestamp=1681034053394Num uniques 1177; 2023-04-09 09:54:13,398 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=53902, new segment data size=8102, suffix heap size=150376,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1177, new segment cells count=177 2023-04-09 09:54:13,400 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681034053392, max timestamp=1681034053394Num uniques 1177; 2023-04-09 09:54:13,400 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=53902, new segment data size=8102, suffix heap size=150376,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1177, new segment cells count=177 2023-04-09 09:54:13,400 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:13,400 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:13,403 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:13,403 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:13,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:13,403 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:13,403 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:13,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:13,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:13,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:13,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:13,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:13,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4258cca9[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:13,406 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:13,406 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:13,408 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:13,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:13,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:13,408 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:13,408 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:13,408 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:13,408 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,408 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:13,409 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:13,409 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:13,409 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,410 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:13,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:13,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:13,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:13,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:13,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:13,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,411 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230409907e8fca30524771b1424e1526db2d8f to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409907e8fca30524771b1424e1526db2d8f 2023-04-09 09:54:13,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:13,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:13,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:13,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:13,413 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:13,413 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:13,413 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:13,413 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:13,413 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:13,415 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:13,415 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:13,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:13,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:13,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:13,415 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:13,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:13,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:13,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:13,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:13,416 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:13,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:13,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:13,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:13,418 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:13,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:13,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:13,421 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:13,421 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:13,421 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,421 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:13,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:13,421 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:13,421 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:13,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:13,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:13,422 DEBUG [Thread-888] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:13,422 DEBUG [Thread-888] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:13,421 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:13,422 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,422 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:13,422 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:13,422 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:13,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:13,422 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:13,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:13,431 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681034053401, max timestamp=1681034053429Num uniques 1036; 2023-04-09 09:54:13,432 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-09 09:54:13,432 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681034053401, max timestamp=1681034053429Num uniques 1036; 2023-04-09 09:54:13,432 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-09 09:54:13,432 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681034053401, max timestamp=1681034053429Num uniques 1036; 2023-04-09 09:54:13,432 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-09 09:54:13,452 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681034053443, max timestamp=1681034053449Num uniques 1095; 2023-04-09 09:54:13,452 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-09 09:54:13,452 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681034053443, max timestamp=1681034053449Num uniques 1095; 2023-04-09 09:54:13,452 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-09 09:54:13,452 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681034053443, max timestamp=1681034053449Num uniques 1095; 2023-04-09 09:54:13,452 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-09 09:54:13,483 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=504, cellSize=23080, totalHeapSize=52656, min timestamp=1681034053461, max timestamp=1681034053480Num uniques 1054; 2023-04-09 09:54:13,483 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=23080, suffix heap size=137584,new segment heap size=52656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=504 2023-04-09 09:54:13,483 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=504, cellSize=23080, totalHeapSize=52656, min timestamp=1681034053461, max timestamp=1681034053480Num uniques 1054; 2023-04-09 09:54:13,483 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=23080, suffix heap size=137584,new segment heap size=52656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=504 2023-04-09 09:54:13,483 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=504, cellSize=23080, totalHeapSize=52656, min timestamp=1681034053461, max timestamp=1681034053480Num uniques 1054; 2023-04-09 09:54:13,484 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=23080, suffix heap size=137584,new segment heap size=52656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=504 2023-04-09 09:54:13,498 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:13,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59766 deadline: 1681034113497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,499 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:13,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59788 deadline: 1681034113498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,500 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:13,500 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:13,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59752 deadline: 1681034113498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59804 deadline: 1681034113499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,501 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:13,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59822 deadline: 1681034113499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,522 DEBUG [Thread-888] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:13,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:13,525 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:13,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:13,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:13,532 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:13,532 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:13,532 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:13,532 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:13,533 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:13,533 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:13,534 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:13,534 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:13,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:13,534 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,534 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:13,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:13,535 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:13,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:13,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:13,536 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:13,540 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:13,540 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:13,540 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:13,540 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:13,540 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:13,552 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:13,552 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:13,560 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:13,560 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. started... 2023-04-09 09:54:13,560 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. as already flushing 2023-04-09 09:54:13,560 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:13,560 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:13,560 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:13,560 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,564 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:13,564 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:13,565 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:13,566 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:13,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:13,566 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:13,566 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:13,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:13,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:13,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:13,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:13,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:13,568 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:13,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f7302ba[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:13,568 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:13,571 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:13,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:13,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:13,571 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:13,571 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:13,571 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:13,571 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,571 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:13,575 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:13,575 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:13,575 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:13,575 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:13,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:13,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:13,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:13,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:13,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:13,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:13,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:13,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:13,579 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:13,579 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:13,579 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:13,579 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:13,579 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:13,581 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:13,581 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:13,581 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:13,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:13,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:13,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:13,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:13,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:13,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:13,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:13,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:13,582 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:13,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:13,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:13,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:13,583 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:13,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:13,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,587 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:13,587 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:13,587 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:13,587 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:13,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:13,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:13,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:13,587 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:13,587 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:13,587 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,587 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:13,587 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:13,587 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:13,587 DEBUG [Thread-888] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:13,587 DEBUG [Thread-888] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:13,587 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:13,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:13,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:13,601 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:13,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59788 deadline: 1681034113601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,606 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:13,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59822 deadline: 1681034113603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,606 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:13,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59766 deadline: 1681034113603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,687 DEBUG [Thread-888] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:13,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:13,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:13,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:13,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:13,692 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:13,692 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:13,692 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:13,692 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:13,693 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:13,693 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:13,694 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,694 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:13,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:13,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:13,695 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,695 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:13,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:13,695 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:13,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:13,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:13,695 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:13,700 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:13,700 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:13,700 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:13,700 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:13,700 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:13,701 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:13,701 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:13,703 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:13,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59804 deadline: 1681034113703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,705 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:13,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59752 deadline: 1681034113704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,708 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:13,708 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. started... 2023-04-09 09:54:13,708 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. as already flushing 2023-04-09 09:54:13,708 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:13,708 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:13,708 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:13,708 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,708 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:13,708 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:13,710 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,710 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:13,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:13,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:13,710 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:13,710 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:13,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:13,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:13,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:13,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:13,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:13,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@547b6cda[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:13,712 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:13,712 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:13,714 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:13,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:13,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:13,715 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:13,715 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:13,715 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:13,715 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,715 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:13,717 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,717 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:13,717 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:13,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:13,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:13,717 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:13,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:13,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:13,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:13,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:13,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:13,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:13,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:13,720 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:13,720 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:13,720 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:13,720 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:13,720 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:13,722 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:13,722 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:13,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:13,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:13,722 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:13,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:13,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:13,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:13,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:13,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:13,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:13,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:13,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:13,724 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:13,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:13,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,727 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:13,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:13,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,729 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,729 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:13,729 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:13,730 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,730 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:13,730 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:13,730 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:13,730 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:13,730 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:13,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:13,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:13,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:13,730 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:13,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:13,730 DEBUG [Thread-888] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:13,730 DEBUG [Thread-888] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:13,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:13,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:13,803 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:13,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59788 deadline: 1681034113803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,810 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:13,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59822 deadline: 1681034113808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,812 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:13,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59766 deadline: 1681034113811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,830 DEBUG [Thread-888] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:13,831 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=38, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/A/e5ea7c3ce89240e99a328e3ed77eadbe 2023-04-09 09:54:13,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:13,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:13,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:13,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:13,860 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:13,860 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:13,860 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:13,860 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:13,861 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:13,861 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:13,863 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:13,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:13,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:13,863 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:13,864 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,864 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:13,864 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:13,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:13,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:13,865 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:13,880 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:13,880 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:13,880 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:13,880 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:13,880 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:13,889 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:13,889 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:13,889 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:13,889 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. started... 2023-04-09 09:54:13,889 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. as already flushing 2023-04-09 09:54:13,889 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:13,889 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:13,889 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:13,889 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,889 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:13,889 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:13,889 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=38 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/B/a19da86c8d2b4922a3309a1d5bceb826 2023-04-09 09:54:13,892 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:13,893 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:13,893 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:13,895 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:13,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:13,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:13,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:13,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:13,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:13,898 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:13,898 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@121aae5f[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:13,898 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:13,898 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:13,899 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:13,900 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:13,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:13,900 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:13,900 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:13,900 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:13,900 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:13,904 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:13,904 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,904 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:13,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:13,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:13,904 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:13,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:13,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:13,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:13,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:13,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:13,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:13,910 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:13,910 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:13,910 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:13,910 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:13,910 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:13,910 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:13,911 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:13,911 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:13,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:13,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:13,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:13,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:13,911 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:13,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:13,912 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:13,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:13,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:13,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:13,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:13,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:13,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:13,914 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:13,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:13,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,922 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:13,922 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:13,922 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:13,922 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,922 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:13,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:13,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:13,922 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:13,922 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:13,922 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:13,922 DEBUG [Thread-888] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:13,922 DEBUG [Thread-888] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:13,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:13,922 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:13,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:13,922 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:13,923 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:13,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:13,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:13,928 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=38 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/C/dc26da56f5794ff0933d644001475b6c 2023-04-09 09:54:13,934 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/A/e5ea7c3ce89240e99a328e3ed77eadbe as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/e5ea7c3ce89240e99a328e3ed77eadbe 2023-04-09 09:54:13,955 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/e5ea7c3ce89240e99a328e3ed77eadbe, entries=150, sequenceid=38, filesize=25.1 K 2023-04-09 09:54:13,956 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/B/a19da86c8d2b4922a3309a1d5bceb826 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/a19da86c8d2b4922a3309a1d5bceb826 2023-04-09 09:54:13,962 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/a19da86c8d2b4922a3309a1d5bceb826, entries=150, sequenceid=38, filesize=11.6 K 2023-04-09 09:54:13,963 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/C/dc26da56f5794ff0933d644001475b6c as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/dc26da56f5794ff0933d644001475b6c 2023-04-09 09:54:13,967 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/dc26da56f5794ff0933d644001475b6c, entries=150, sequenceid=38, filesize=11.6 K 2023-04-09 09:54:13,968 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~86.11 KB/88176, currentSize=161.02 KB/164880 for b86d94f6f111f1359f84891af8d12a40 in 652ms, sequenceid=38, compaction requested=false 2023-04-09 09:54:13,968 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b86d94f6f111f1359f84891af8d12a40: 2023-04-09 09:54:14,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on b86d94f6f111f1359f84891af8d12a40 2023-04-09 09:54:14,010 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b86d94f6f111f1359f84891af8d12a40 3/3 column families, dataSize=167.72 KB heapSize=452.25 KB 2023-04-09 09:54:14,010 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=A 2023-04-09 09:54:14,010 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:54:14,010 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=B 2023-04-09 09:54:14,010 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:54:14,010 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=C 2023-04-09 09:54:14,010 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:54:14,016 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:14,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59804 deadline: 1681034114013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,018 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:14,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59752 deadline: 1681034114016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,022 DEBUG [Thread-888] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:14,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:14,026 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:14,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:14,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:14,040 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:14,040 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:14,040 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:14,041 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:14,041 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040964f7b59a1490487481756428ea877142 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040964f7b59a1490487481756428ea877142 2023-04-09 09:54:14,041 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:14,041 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:14,043 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:14,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:14,043 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:14,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:14,043 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,043 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:14,044 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:14,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:14,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:14,045 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:14,050 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:14,050 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:14,050 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:14,051 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:14,051 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:14,053 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:14,053 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:14,053 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:14,053 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. started... 2023-04-09 09:54:14,053 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. as already flushing 2023-04-09 09:54:14,053 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:14,053 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:14,053 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:14,053 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,053 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:14,053 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:14,060 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,060 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:14,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:14,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:14,061 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:14,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:14,061 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:14,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:14,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:14,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:14,064 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:14,064 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:14,064 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16230531[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:14,064 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:14,065 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:14,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:14,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:14,066 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:14,066 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:14,066 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:14,066 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,066 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:14,068 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,068 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:14,068 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:14,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:14,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:14,068 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:14,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:14,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:14,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:14,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:14,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:14,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:14,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:14,072 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:14,072 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:14,072 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:14,072 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:14,072 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:14,076 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:14,076 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:14,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:14,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:14,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:14,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:14,076 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:14,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:14,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:14,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:14,077 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:14,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:14,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:14,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:14,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:14,078 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:14,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:14,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,085 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,085 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:14,085 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:14,085 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:14,085 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:14,085 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:14,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:14,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:14,085 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:14,085 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:14,085 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,085 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:14,085 DEBUG [Thread-888] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:14,086 DEBUG [Thread-888] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:14,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:14,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:14,085 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:14,086 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:14,106 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:14,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59788 deadline: 1681034114105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,113 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:14,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59822 deadline: 1681034114112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,114 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:14,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59766 deadline: 1681034114113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,119 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:14,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59804 deadline: 1681034114117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,120 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:14,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59752 deadline: 1681034114120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,186 DEBUG [Thread-888] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:14,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:14,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:14,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:14,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:14,194 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:14,194 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:14,194 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:14,194 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:14,194 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:14,194 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:14,197 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:14,197 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:14,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:14,197 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,197 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:14,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:14,198 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:14,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:14,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:14,198 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:14,212 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:14,212 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:14,212 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:14,212 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:14,212 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:14,232 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:14,232 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:14,232 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:14,232 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. started... 2023-04-09 09:54:14,232 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. as already flushing 2023-04-09 09:54:14,232 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:14,232 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:14,232 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:14,232 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,232 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:14,232 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:14,234 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,234 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:14,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:14,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:14,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:14,235 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:14,235 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:14,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:14,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:14,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:14,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:14,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d2a39d[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:14,237 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:14,237 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:14,239 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:14,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:14,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:14,239 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,239 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:14,240 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:14,240 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:14,240 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:14,241 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:14,241 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:14,241 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,241 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:14,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:14,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:14,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:14,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:14,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:14,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:14,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:14,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:14,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:14,247 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:14,247 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:14,247 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:14,247 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:14,247 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:14,250 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:14,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:14,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:14,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:14,250 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:14,251 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:14,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:14,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:14,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:14,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:14,251 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:14,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:14,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:14,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:14,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:14,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:14,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:14,254 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,262 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:14,262 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:14,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:14,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:14,263 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,263 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:14,263 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:14,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:14,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:14,263 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,263 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:14,263 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:14,263 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:14,263 DEBUG [Thread-888] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:14,263 DEBUG [Thread-888] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:14,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:14,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:14,263 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:14,323 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:14,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59804 deadline: 1681034114322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,326 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:14,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59752 deadline: 1681034114323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,364 DEBUG [Thread-888] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:14,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:14,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:14,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:14,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:14,370 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:14,370 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:14,370 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:14,370 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:14,371 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:14,371 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:14,376 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,376 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:14,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:14,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:14,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:14,377 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,377 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:14,378 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:14,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:14,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:14,378 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:14,380 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:14,380 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:14,380 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:14,380 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:14,380 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:14,390 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:14,390 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:14,390 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:14,390 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. started... 2023-04-09 09:54:14,390 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. as already flushing 2023-04-09 09:54:14,390 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:14,390 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:14,390 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:14,390 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,391 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:14,391 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:14,393 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,393 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:14,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:14,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:14,394 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:14,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:14,394 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:14,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:14,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:14,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:14,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:14,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e4002c1[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:14,395 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:14,396 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:14,397 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:14,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:14,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:14,398 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:14,398 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:14,398 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:14,399 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,399 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:14,399 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:14,399 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:14,399 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:14,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:14,399 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:14,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:14,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:14,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:14,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:14,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:14,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:14,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:14,404 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:14,404 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:14,404 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:14,404 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:14,404 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:14,406 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:14,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:14,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:14,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:14,406 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:14,406 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:14,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:14,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:14,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:14,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:14,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:14,408 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:14,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:14,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:14,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:14,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,410 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:14,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:14,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,412 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:14,413 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:14,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:14,413 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,413 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:14,413 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:14,413 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:14,413 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,413 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:14,413 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:14,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:14,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:14,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:14,413 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:14,413 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:14,413 DEBUG [Thread-888] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:14,413 DEBUG [Thread-888] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:14,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:14,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:14,456 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=159, memsize=55.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/A/3359657996a64dd8b74c07c96c1c33d6 2023-04-09 09:54:14,480 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=55.91 KB at sequenceid=159 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/B/5af34ea9f9e04bf3b298b87dec4719ad 2023-04-09 09:54:14,510 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=55.91 KB at sequenceid=159 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/C/6b25e4ea2bd94e0c81ce6e6b2e95e215 2023-04-09 09:54:14,513 DEBUG [Thread-888] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:14,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:14,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:14,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:14,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:14,521 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:14,521 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:14,521 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:14,521 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:14,522 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:14,522 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:14,529 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/A/3359657996a64dd8b74c07c96c1c33d6 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/3359657996a64dd8b74c07c96c1c33d6 2023-04-09 09:54:14,529 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:14,529 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:14,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:14,530 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:14,530 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:14,531 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:14,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:14,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:14,531 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:14,532 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:14,532 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:14,532 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:14,532 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:14,532 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:14,535 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/3359657996a64dd8b74c07c96c1c33d6, entries=150, sequenceid=159, filesize=25.3 K 2023-04-09 09:54:14,537 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:14,537 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:14,538 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/B/5af34ea9f9e04bf3b298b87dec4719ad as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/5af34ea9f9e04bf3b298b87dec4719ad 2023-04-09 09:54:14,545 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/5af34ea9f9e04bf3b298b87dec4719ad, entries=150, sequenceid=159, filesize=11.8 K 2023-04-09 09:54:14,545 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:14,545 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. started... 2023-04-09 09:54:14,546 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. as already flushing 2023-04-09 09:54:14,546 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:14,546 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:14,546 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:14,546 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,546 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/C/6b25e4ea2bd94e0c81ce6e6b2e95e215 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/6b25e4ea2bd94e0c81ce6e6b2e95e215 2023-04-09 09:54:14,546 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:14,546 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:14,548 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:14,548 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:14,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:14,548 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:14,548 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:14,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:14,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:14,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:14,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:14,550 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:14,550 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@496f2d00[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:14,550 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:14,550 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:14,552 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:14,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:14,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:14,552 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:14,552 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:14,552 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:14,553 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,553 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:14,553 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/6b25e4ea2bd94e0c81ce6e6b2e95e215, entries=150, sequenceid=159, filesize=11.8 K 2023-04-09 09:54:14,554 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,554 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:14,554 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:14,554 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:14,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:14,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:14,554 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~167.72 KB/171750, heapSize ~452.20 KB/463056, currentSize=20.13 KB/20610 for b86d94f6f111f1359f84891af8d12a40 in 544ms, sequenceid=159, compaction requested=true 2023-04-09 09:54:14,554 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b86d94f6f111f1359f84891af8d12a40: 2023-04-09 09:54:14,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:14,555 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:54:14,555 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-09 09:54:14,555 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:54:14,555 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-09 09:54:14,555 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:54:14,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:14,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:14,556 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 77388 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-09 09:54:14,556 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35808 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-09 09:54:14,557 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1912): b86d94f6f111f1359f84891af8d12a40/A is initiating minor compaction (all files) 2023-04-09 09:54:14,557 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): b86d94f6f111f1359f84891af8d12a40/B is initiating minor compaction (all files) 2023-04-09 09:54:14,557 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b86d94f6f111f1359f84891af8d12a40/A in TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:14,557 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b86d94f6f111f1359f84891af8d12a40/B in TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:14,557 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/197873c5abc04c129b4e40349c8944c5, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/e5ea7c3ce89240e99a328e3ed77eadbe, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/3359657996a64dd8b74c07c96c1c33d6] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp, totalSize=75.6 K 2023-04-09 09:54:14,557 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/4f639fd99d8f423c8021f57d15986701, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/a19da86c8d2b4922a3309a1d5bceb826, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/5af34ea9f9e04bf3b298b87dec4719ad] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp, totalSize=35.0 K 2023-04-09 09:54:14,557 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 197873c5abc04c129b4e40349c8944c5, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1681034052972 2023-04-09 09:54:14,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:14,558 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting e5ea7c3ce89240e99a328e3ed77eadbe, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=38, earliestPutTs=1681034053001 2023-04-09 09:54:14,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:14,558 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 4f639fd99d8f423c8021f57d15986701, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1681034052972 2023-04-09 09:54:14,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,559 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 3359657996a64dd8b74c07c96c1c33d6, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=159, earliestPutTs=1681034053496 2023-04-09 09:54:14,559 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting a19da86c8d2b4922a3309a1d5bceb826, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=38, earliestPutTs=1681034053001 2023-04-09 09:54:14,559 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 5af34ea9f9e04bf3b298b87dec4719ad, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=159, earliestPutTs=1681034053496 2023-04-09 09:54:14,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:14,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:14,560 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:14,560 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:14,560 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:14,560 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:14,560 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:14,563 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:14,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:14,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:14,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:14,563 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:14,563 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:14,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:14,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:14,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:14,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:14,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:14,564 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:14,566 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:14,569 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:14,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:14,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:14,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:14,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:14,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:14,569 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,569 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:14,569 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:14,569 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,569 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:14,569 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:14,569 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:14,569 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:14,569 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:14,569 DEBUG [Thread-888] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:14,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:14,570 DEBUG [Thread-888] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:14,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:14,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:14,570 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:14,588 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): b86d94f6f111f1359f84891af8d12a40#B#compaction#124 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:54:14,598 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] throttle.PressureAwareThroughputController(145): b86d94f6f111f1359f84891af8d12a40#A#compaction#123 average throughput is 0.90 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:54:14,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on b86d94f6f111f1359f84891af8d12a40 2023-04-09 09:54:14,621 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b86d94f6f111f1359f84891af8d12a40 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-09 09:54:14,621 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=A 2023-04-09 09:54:14,621 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:14,622 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=B 2023-04-09 09:54:14,622 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:14,622 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=C 2023-04-09 09:54:14,622 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:14,633 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/B/15f2f6b0d9f94eaabb88ac1b4c29d88e as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/15f2f6b0d9f94eaabb88ac1b4c29d88e 2023-04-09 09:54:14,643 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b86d94f6f111f1359f84891af8d12a40/B of b86d94f6f111f1359f84891af8d12a40 into 15f2f6b0d9f94eaabb88ac1b4c29d88e(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:54:14,643 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b86d94f6f111f1359f84891af8d12a40: 2023-04-09 09:54:14,644 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40., storeName=b86d94f6f111f1359f84891af8d12a40/B, priority=13, startTime=1681034054555; duration=0sec 2023-04-09 09:54:14,644 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:54:14,644 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-09 09:54:14,645 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35808 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-09 09:54:14,645 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): b86d94f6f111f1359f84891af8d12a40/C is initiating minor compaction (all files) 2023-04-09 09:54:14,646 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b86d94f6f111f1359f84891af8d12a40/C in TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:14,646 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/212195da92fc4eaeadc2aa3ef754b12a, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/dc26da56f5794ff0933d644001475b6c, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/6b25e4ea2bd94e0c81ce6e6b2e95e215] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp, totalSize=35.0 K 2023-04-09 09:54:14,646 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 212195da92fc4eaeadc2aa3ef754b12a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1681034052972 2023-04-09 09:54:14,647 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting dc26da56f5794ff0933d644001475b6c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=38, earliestPutTs=1681034053001 2023-04-09 09:54:14,647 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 6b25e4ea2bd94e0c81ce6e6b2e95e215, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=159, earliestPutTs=1681034053496 2023-04-09 09:54:14,656 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681034054636, max timestamp=1681034054651Num uniques 859; 2023-04-09 09:54:14,656 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-09 09:54:14,656 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): b86d94f6f111f1359f84891af8d12a40#C#compaction#126 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:54:14,657 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681034054636, max timestamp=1681034054651Num uniques 859; 2023-04-09 09:54:14,657 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230409b6428e591f6641fdb831f2362703c1c8 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409b6428e591f6641fdb831f2362703c1c8 2023-04-09 09:54:14,658 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-09 09:54:14,658 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681034054636, max timestamp=1681034054651Num uniques 859; 2023-04-09 09:54:14,658 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-09 09:54:14,670 DEBUG [Thread-888] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:14,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:14,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:14,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:14,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:14,677 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:14,677 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:14,677 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:14,677 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:14,678 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:14,678 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:14,678 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=169, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/A/95894e58446548c1b168ffa29667e33c 2023-04-09 09:54:14,680 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:14,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:14,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:14,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:14,680 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,681 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:14,681 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,681 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:14,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:14,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:14,681 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:14,681 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:14,682 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:14,682 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:14,682 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:14,682 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:14,682 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:14,682 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:14,682 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:14,682 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. started... 2023-04-09 09:54:14,682 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. as already flushing 2023-04-09 09:54:14,682 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:14,682 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:14,682 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:14,682 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,683 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:14,683 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:14,687 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:14,687 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:14,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:14,687 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:14,687 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:14,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:14,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:14,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:14,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,688 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:14,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59752 deadline: 1681034114676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,689 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:14,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:14,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59822 deadline: 1681034114680, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,689 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:14,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59788 deadline: 1681034114680, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,689 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:14,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59804 deadline: 1681034114688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:14,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47ebb391[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:14,692 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:14,692 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:14,693 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:14,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:14,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:14,694 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:14,694 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:14,694 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:14,695 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:14,695 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,695 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:14,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59766 deadline: 1681034114690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,697 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:14,697 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:14,697 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:14,697 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:14,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:14,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:14,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:14,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:14,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:14,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:14,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:14,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:14,701 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:14,701 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:14,701 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:14,701 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:14,701 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:14,702 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:14,702 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:14,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:14,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:14,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:14,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:14,702 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:14,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:14,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:14,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:14,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:14,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:14,704 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:14,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:14,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:14,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,706 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:14,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:14,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,710 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:14,710 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:14,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:14,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:14,710 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,710 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:14,710 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:14,710 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:14,710 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:14,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:14,710 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:14,711 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:14,711 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:14,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:14,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:14,712 DEBUG [Thread-888] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:14,712 DEBUG [Thread-888] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:14,720 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/C/d3c10f3f711f4c979980e64f9a039b3b as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/d3c10f3f711f4c979980e64f9a039b3b 2023-04-09 09:54:14,733 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b86d94f6f111f1359f84891af8d12a40/C of b86d94f6f111f1359f84891af8d12a40 into d3c10f3f711f4c979980e64f9a039b3b(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:54:14,733 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b86d94f6f111f1359f84891af8d12a40: 2023-04-09 09:54:14,733 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40., storeName=b86d94f6f111f1359f84891af8d12a40/C, priority=13, startTime=1681034054555; duration=0sec 2023-04-09 09:54:14,733 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:54:14,791 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:14,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59752 deadline: 1681034114790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,792 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:14,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59804 deadline: 1681034114791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,793 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:14,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59822 deadline: 1681034114791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,793 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:14,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59788 deadline: 1681034114791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,800 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:14,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59766 deadline: 1681034114797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,812 DEBUG [Thread-888] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:14,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:14,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:14,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:14,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:14,823 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:14,823 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:14,823 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:14,823 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:14,824 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:14,824 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:14,825 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:14,825 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:14,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:14,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:14,826 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:14,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:14,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:14,827 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:14,832 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,832 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:14,834 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:14,834 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:14,834 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:14,834 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:14,835 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:14,836 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:14,836 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:14,836 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:14,836 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. started... 2023-04-09 09:54:14,837 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. as already flushing 2023-04-09 09:54:14,837 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:14,837 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:14,837 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:14,837 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,837 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:14,837 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:14,838 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,838 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:14,839 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:14,839 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:14,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:14,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:14,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:14,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:14,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:14,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:14,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:14,842 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:14,842 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:14,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53b13eba[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:14,843 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:14,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:14,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:14,843 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:14,843 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:14,843 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:14,844 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,844 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:14,846 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:14,846 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:14,846 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,846 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:14,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:14,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:14,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:14,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:14,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:14,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:14,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:14,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:14,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:14,851 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:14,851 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:14,851 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:14,851 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:14,851 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:14,852 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:14,852 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:14,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:14,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:14,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:14,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:14,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:14,852 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:14,853 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:14,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:14,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:14,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:14,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:14,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:14,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:14,854 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:14,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:14,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,856 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,856 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:14,856 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:14,856 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:14,856 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:14,856 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:14,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:14,856 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:14,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:14,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:14,856 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:14,856 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:14,856 DEBUG [Thread-888] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:14,856 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:14,856 DEBUG [Thread-888] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:14,857 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:14,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:14,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:14,930 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-09 09:54:14,963 DEBUG [Thread-888] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:14,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:14,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:14,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:14,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:14,969 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:14,969 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:14,969 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:14,969 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:14,970 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:14,970 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:14,972 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:14,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:14,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:14,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:14,973 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,973 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:14,973 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:14,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:14,973 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:14,973 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:14,974 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:14,974 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:14,974 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:14,974 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:14,974 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:14,974 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:14,974 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:14,974 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. started... 2023-04-09 09:54:14,975 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. as already flushing 2023-04-09 09:54:14,975 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:14,975 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:14,975 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:14,975 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,975 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:14,975 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:14,976 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,976 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:14,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:14,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:14,977 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:14,977 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:14,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:14,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:14,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:14,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:14,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:14,979 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:14,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d697807[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:14,979 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:14,980 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:14,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:14,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:14,980 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:14,980 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:14,980 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,980 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:14,980 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:14,983 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,983 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:14,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:14,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:14,983 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:14,983 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:14,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:14,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:14,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:14,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:14,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:14,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:14,986 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:14,986 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:14,987 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:14,987 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:14,987 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:14,987 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:14,989 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:14,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:14,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:14,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:14,989 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:14,989 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:14,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:14,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:14,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:14,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:14,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:14,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:14,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:14,991 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:14,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:14,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:14,992 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:14,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,997 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:14,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59752 deadline: 1681034114993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,998 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:14,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59788 deadline: 1681034114994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,998 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,998 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:14,998 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:14,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59804 deadline: 1681034114995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:14,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:14,998 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:14,998 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:14,999 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,999 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:14,999 DEBUG [Thread-888] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:14,999 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:14,999 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:14,999 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:14,999 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:14,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59822 deadline: 1681034114996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:14,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:14,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:14,999 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:14,999 DEBUG [Thread-888] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:15,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:15,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:15,007 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:15,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59766 deadline: 1681034115004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:15,065 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/A/6165fcd6cc284b26838e6a027e142680 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/6165fcd6cc284b26838e6a027e142680 2023-04-09 09:54:15,071 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b86d94f6f111f1359f84891af8d12a40/A of b86d94f6f111f1359f84891af8d12a40 into 6165fcd6cc284b26838e6a027e142680(size=25.4 K), total size for store is 25.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:54:15,071 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b86d94f6f111f1359f84891af8d12a40: 2023-04-09 09:54:15,071 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40., storeName=b86d94f6f111f1359f84891af8d12a40/A, priority=13, startTime=1681034054555; duration=0sec 2023-04-09 09:54:15,071 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:54:15,099 DEBUG [Thread-888] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:15,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:15,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:15,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:15,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:15,103 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:15,103 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:15,103 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:15,103 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:15,104 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:15,104 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:15,105 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:15,105 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:15,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:15,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:15,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:15,106 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:15,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:15,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:15,107 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:15,107 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:15,107 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:15,107 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:15,107 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:15,107 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:15,107 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:15,108 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:15,108 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:15,108 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. started... 2023-04-09 09:54:15,108 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:15,108 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:15,108 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. as already flushing 2023-04-09 09:54:15,108 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:15,108 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:15,108 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:15,108 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:15,108 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:15,108 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:15,110 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:15,110 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:15,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:15,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:15,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:15,110 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:15,110 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:15,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:15,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:15,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:15,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:15,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:15,111 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:15,111 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:15,111 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:15,111 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6dde5987[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:15,112 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:15,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:15,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:15,112 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:15,112 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:15,112 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:15,113 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:15,113 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:15,114 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:15,114 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:15,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:15,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:15,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:15,114 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:15,114 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:15,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:15,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:15,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:15,115 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=169 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/B/6c8ea589c84e492eb6256ef7a9ae985a 2023-04-09 09:54:15,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:15,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:15,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:15,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:15,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:15,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:15,118 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:15,118 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:15,118 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:15,118 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:15,118 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:15,121 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:15,121 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:15,121 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:15,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:15,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:15,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:15,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:15,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:15,121 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:15,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:15,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:15,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:15,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:15,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:15,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:15,122 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:15,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:15,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:15,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:15,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:15,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:15,125 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:15,125 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:15,125 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:15,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:15,125 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:15,125 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:15,125 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:15,125 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:15,125 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:15,125 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:15,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:15,125 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:15,125 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:15,125 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:15,125 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:15,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:15,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:15,125 DEBUG [Thread-888] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:15,126 DEBUG [Thread-888] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:15,131 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=169 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/C/2c38eaa56c724348b2a4338fb35d9e4d 2023-04-09 09:54:15,136 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/A/95894e58446548c1b168ffa29667e33c as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/95894e58446548c1b168ffa29667e33c 2023-04-09 09:54:15,140 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/95894e58446548c1b168ffa29667e33c, entries=250, sequenceid=169, filesize=38.9 K 2023-04-09 09:54:15,141 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/B/6c8ea589c84e492eb6256ef7a9ae985a as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/6c8ea589c84e492eb6256ef7a9ae985a 2023-04-09 09:54:15,147 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/6c8ea589c84e492eb6256ef7a9ae985a, entries=150, sequenceid=169, filesize=11.8 K 2023-04-09 09:54:15,149 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/C/2c38eaa56c724348b2a4338fb35d9e4d as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/2c38eaa56c724348b2a4338fb35d9e4d 2023-04-09 09:54:15,156 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/2c38eaa56c724348b2a4338fb35d9e4d, entries=150, sequenceid=169, filesize=11.8 K 2023-04-09 09:54:15,157 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for b86d94f6f111f1359f84891af8d12a40 in 536ms, sequenceid=169, compaction requested=false 2023-04-09 09:54:15,157 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b86d94f6f111f1359f84891af8d12a40: 2023-04-09 09:54:15,226 DEBUG [Thread-888] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:15,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:15,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:15,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:15,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:15,229 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:15,229 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:15,230 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:15,230 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:15,230 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:15,230 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:15,231 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:15,231 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:15,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:15,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:15,231 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:15,232 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:15,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:15,232 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:15,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:15,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:15,233 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:15,233 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:15,233 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:15,233 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:15,233 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:15,233 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:15,234 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:15,234 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:15,234 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:15,234 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. started... 2023-04-09 09:54:15,234 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b86d94f6f111f1359f84891af8d12a40 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-09 09:54:15,234 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=A 2023-04-09 09:54:15,234 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:54:15,234 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=B 2023-04-09 09:54:15,234 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:54:15,234 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=C 2023-04-09 09:54:15,234 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:54:15,257 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040968230600d4714275ba0d1cf24fed7fad to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040968230600d4714275ba0d1cf24fed7fad 2023-04-09 09:54:15,266 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=209, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/A/00795246b54341b0a7418e8de32740b2 2023-04-09 09:54:15,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on b86d94f6f111f1359f84891af8d12a40 2023-04-09 09:54:15,301 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. as already flushing 2023-04-09 09:54:15,307 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:15,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59804 deadline: 1681034115303, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:15,309 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:15,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59822 deadline: 1681034115306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:15,310 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:15,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59752 deadline: 1681034115307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:15,311 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:15,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59788 deadline: 1681034115307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:15,311 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:15,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59766 deadline: 1681034115309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:15,411 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:15,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59804 deadline: 1681034115408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:15,411 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:15,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59822 deadline: 1681034115410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:15,412 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:15,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59752 deadline: 1681034115412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:15,412 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:15,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59788 deadline: 1681034115412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:15,616 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:15,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59788 deadline: 1681034115613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:15,616 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:15,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59804 deadline: 1681034115614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:15,617 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:15,616 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:15,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59822 deadline: 1681034115614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:15,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59752 deadline: 1681034115614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:15,688 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=209 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/B/5ff55b9f560b44e48bdccde66d20d3b5 2023-04-09 09:54:15,715 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=209 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/C/f58f3d0e0235434d92830845f1a89175 2023-04-09 09:54:15,722 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/A/00795246b54341b0a7418e8de32740b2 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/00795246b54341b0a7418e8de32740b2 2023-04-09 09:54:15,727 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/00795246b54341b0a7418e8de32740b2, entries=150, sequenceid=209, filesize=25.3 K 2023-04-09 09:54:15,730 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/B/5ff55b9f560b44e48bdccde66d20d3b5 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/5ff55b9f560b44e48bdccde66d20d3b5 2023-04-09 09:54:15,744 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/5ff55b9f560b44e48bdccde66d20d3b5, entries=150, sequenceid=209, filesize=11.8 K 2023-04-09 09:54:15,745 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/C/f58f3d0e0235434d92830845f1a89175 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/f58f3d0e0235434d92830845f1a89175 2023-04-09 09:54:15,751 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/f58f3d0e0235434d92830845f1a89175, entries=150, sequenceid=209, filesize=11.8 K 2023-04-09 09:54:15,752 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=46.96 KB/48090 for b86d94f6f111f1359f84891af8d12a40 in 518ms, sequenceid=209, compaction requested=true 2023-04-09 09:54:15,752 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b86d94f6f111f1359f84891af8d12a40: 2023-04-09 09:54:15,752 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:15,752 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:15,752 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:15,752 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:15,752 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:15,752 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:15,754 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:15,754 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:15,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:15,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:15,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:15,754 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:15,754 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:15,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:15,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:15,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:15,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:15,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:15,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:15,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77b30896[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:15,756 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:15,756 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:15,757 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:15,757 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:15,759 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:15,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:15,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:15,759 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:15,759 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:15,759 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:15,762 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:15,762 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:15,762 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:15,762 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:15,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:15,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:15,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:15,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:15,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:15,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:15,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:15,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:15,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:15,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:15,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:15,767 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:15,767 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:15,767 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:15,767 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:15,767 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:15,767 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:15,770 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:15,770 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:15,770 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:15,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:15,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:15,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:15,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:15,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:15,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:15,772 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:15,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:15,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:15,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:15,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:15,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:15,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:15,773 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:15,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:15,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:15,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:15,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:15,775 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:15,775 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:15,775 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:15,775 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:15,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:15,775 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:15,775 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:15,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:15,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:15,775 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:15,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:15,776 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:15,776 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:15,776 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:15,776 DEBUG [Thread-888] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:15,776 DEBUG [Thread-888] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:15,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:15,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:15,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on b86d94f6f111f1359f84891af8d12a40 2023-04-09 09:54:15,819 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b86d94f6f111f1359f84891af8d12a40 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-09 09:54:15,819 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=A 2023-04-09 09:54:15,819 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:15,819 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=B 2023-04-09 09:54:15,819 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:15,819 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=C 2023-04-09 09:54:15,819 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:15,838 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230409261b98419c1248368e89c746c337efc2 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409261b98419c1248368e89c746c337efc2 2023-04-09 09:54:15,862 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=220, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/A/cf072f3660e1490087a0523afd623ff8 2023-04-09 09:54:15,876 DEBUG [Thread-888] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:15,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:15,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:15,899 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681034055865, max timestamp=1681034055897Num uniques 859; 2023-04-09 09:54:15,899 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-09 09:54:15,899 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681034055865, max timestamp=1681034055897Num uniques 859; 2023-04-09 09:54:15,899 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-09 09:54:15,899 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681034055865, max timestamp=1681034055897Num uniques 859; 2023-04-09 09:54:15,899 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-09 09:54:15,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:15,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:15,901 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:15,901 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:15,901 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:15,901 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:15,901 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:15,901 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:15,904 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:15,904 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:15,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:15,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:15,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:15,908 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:15,909 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:15,909 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:15,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:15,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:15,909 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:15,916 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:15,916 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:15,916 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:15,916 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:15,916 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:15,917 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=220 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/B/ee4ed16464f04805b19641b5042620ff 2023-04-09 09:54:15,920 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:15,920 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:15,920 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:15,920 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. started... 2023-04-09 09:54:15,921 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. as already flushing 2023-04-09 09:54:15,921 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:15,921 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:15,921 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:15,921 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:15,921 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:15,921 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:15,924 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:15,924 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:15,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:15,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:15,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:15,924 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:15,924 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:15,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:15,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:15,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:15,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:15,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:15,928 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:15,928 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2dadbc34[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:15,928 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:15,928 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:15,929 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:15,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:15,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:15,929 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:15,930 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:15,930 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:15,930 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:15,930 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:15,932 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:15,932 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:15,932 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:15,932 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:15,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:15,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:15,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:15,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:15,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:15,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:15,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:15,938 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:15,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59822 deadline: 1681034115931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:15,938 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:15,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59752 deadline: 1681034115933, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:15,938 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:15,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59788 deadline: 1681034115935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:15,939 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:15,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59804 deadline: 1681034115935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:15,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:15,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:15,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:15,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:15,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:15,941 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:15,941 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:15,941 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:15,941 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:15,941 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:15,946 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:15,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59766 deadline: 1681034115941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:15,946 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:15,946 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:15,946 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:15,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:15,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:15,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:15,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:15,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:15,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:15,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:15,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:15,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:15,948 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:15,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:15,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:15,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:15,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:15,950 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:15,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:15,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:15,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:15,952 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:15,953 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:15,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:15,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:15,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:15,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:15,953 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:15,954 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:15,954 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:15,954 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:15,954 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:15,954 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:15,954 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:15,954 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:15,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:15,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:15,954 DEBUG [Thread-888] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:15,954 DEBUG [Thread-888] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:15,975 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=220 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/C/d75448fdda2249adb066682e468cfdc6 2023-04-09 09:54:15,983 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/A/cf072f3660e1490087a0523afd623ff8 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/cf072f3660e1490087a0523afd623ff8 2023-04-09 09:54:15,989 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/cf072f3660e1490087a0523afd623ff8, entries=150, sequenceid=220, filesize=25.3 K 2023-04-09 09:54:15,991 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/B/ee4ed16464f04805b19641b5042620ff as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/ee4ed16464f04805b19641b5042620ff 2023-04-09 09:54:15,998 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/ee4ed16464f04805b19641b5042620ff, entries=150, sequenceid=220, filesize=11.8 K 2023-04-09 09:54:15,999 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/C/d75448fdda2249adb066682e468cfdc6 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/d75448fdda2249adb066682e468cfdc6 2023-04-09 09:54:16,012 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/d75448fdda2249adb066682e468cfdc6, entries=150, sequenceid=220, filesize=11.8 K 2023-04-09 09:54:16,013 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for b86d94f6f111f1359f84891af8d12a40 in 194ms, sequenceid=220, compaction requested=true 2023-04-09 09:54:16,014 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b86d94f6f111f1359f84891af8d12a40: 2023-04-09 09:54:16,014 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-09 09:54:16,014 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:54:16,014 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-09 09:54:16,015 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:54:16,015 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:54:16,016 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 117671 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-09 09:54:16,016 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1912): b86d94f6f111f1359f84891af8d12a40/A is initiating minor compaction (all files) 2023-04-09 09:54:16,016 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b86d94f6f111f1359f84891af8d12a40/A in TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:16,016 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/6165fcd6cc284b26838e6a027e142680, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/95894e58446548c1b168ffa29667e33c, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/00795246b54341b0a7418e8de32740b2, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/cf072f3660e1490087a0523afd623ff8] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp, totalSize=114.9 K 2023-04-09 09:54:16,016 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48247 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-09 09:54:16,016 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): b86d94f6f111f1359f84891af8d12a40/B is initiating minor compaction (all files) 2023-04-09 09:54:16,016 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b86d94f6f111f1359f84891af8d12a40/B in TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:16,017 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/15f2f6b0d9f94eaabb88ac1b4c29d88e, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/6c8ea589c84e492eb6256ef7a9ae985a, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/5ff55b9f560b44e48bdccde66d20d3b5, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/ee4ed16464f04805b19641b5042620ff] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp, totalSize=47.1 K 2023-04-09 09:54:16,018 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 15f2f6b0d9f94eaabb88ac1b4c29d88e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=159, earliestPutTs=1681034053496 2023-04-09 09:54:16,018 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 6165fcd6cc284b26838e6a027e142680, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=159, earliestPutTs=1681034053496 2023-04-09 09:54:16,018 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 6c8ea589c84e492eb6256ef7a9ae985a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1681034054014 2023-04-09 09:54:16,021 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 95894e58446548c1b168ffa29667e33c, keycount=250, bloomtype=ROW, size=38.9 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1681034054014 2023-04-09 09:54:16,021 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 5ff55b9f560b44e48bdccde66d20d3b5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1681034054674 2023-04-09 09:54:16,021 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 00795246b54341b0a7418e8de32740b2, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1681034054674 2023-04-09 09:54:16,022 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting ee4ed16464f04805b19641b5042620ff, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=220, earliestPutTs=1681034055305 2023-04-09 09:54:16,022 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting cf072f3660e1490087a0523afd623ff8, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=220, earliestPutTs=1681034055305 2023-04-09 09:54:16,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on b86d94f6f111f1359f84891af8d12a40 2023-04-09 09:54:16,046 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b86d94f6f111f1359f84891af8d12a40 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-09 09:54:16,047 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=A 2023-04-09 09:54:16,047 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:54:16,047 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=B 2023-04-09 09:54:16,047 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): b86d94f6f111f1359f84891af8d12a40#B#compaction#136 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:54:16,047 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:54:16,047 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=C 2023-04-09 09:54:16,048 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:54:16,054 DEBUG [Thread-888] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:16,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:16,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:16,063 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:16,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59766 deadline: 1681034116055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,066 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] throttle.PressureAwareThroughputController(145): b86d94f6f111f1359f84891af8d12a40#A#compaction#135 average throughput is 0.76 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:54:16,068 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:16,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59752 deadline: 1681034116060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,068 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:16,068 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:16,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59822 deadline: 1681034116061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,069 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:16,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59788 deadline: 1681034116063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59804 deadline: 1681034116062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:16,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:16,079 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:16,079 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:16,079 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:16,079 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:16,079 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:16,079 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:16,081 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,082 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:16,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:16,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:16,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:16,082 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,082 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:16,083 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:16,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:16,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:16,083 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:16,088 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:16,088 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:16,088 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:16,088 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:16,088 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:16,105 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/B/25adbe40e82e4215a301c73de4368b9f as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/25adbe40e82e4215a301c73de4368b9f 2023-04-09 09:54:16,112 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b86d94f6f111f1359f84891af8d12a40/B of b86d94f6f111f1359f84891af8d12a40 into 25adbe40e82e4215a301c73de4368b9f(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:54:16,112 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b86d94f6f111f1359f84891af8d12a40: 2023-04-09 09:54:16,112 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40., storeName=b86d94f6f111f1359f84891af8d12a40/B, priority=12, startTime=1681034056014; duration=0sec 2023-04-09 09:54:16,112 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:54:16,112 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-09 09:54:16,114 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48247 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-09 09:54:16,114 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): b86d94f6f111f1359f84891af8d12a40/C is initiating minor compaction (all files) 2023-04-09 09:54:16,114 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b86d94f6f111f1359f84891af8d12a40/C in TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:16,114 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/d3c10f3f711f4c979980e64f9a039b3b, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/2c38eaa56c724348b2a4338fb35d9e4d, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/f58f3d0e0235434d92830845f1a89175, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/d75448fdda2249adb066682e468cfdc6] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp, totalSize=47.1 K 2023-04-09 09:54:16,115 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting d3c10f3f711f4c979980e64f9a039b3b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=159, earliestPutTs=1681034053496 2023-04-09 09:54:16,116 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:16,116 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:16,116 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:16,116 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 2c38eaa56c724348b2a4338fb35d9e4d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1681034054014 2023-04-09 09:54:16,125 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. started... 2023-04-09 09:54:16,126 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting f58f3d0e0235434d92830845f1a89175, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1681034054674 2023-04-09 09:54:16,126 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. as already flushing 2023-04-09 09:54:16,127 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting d75448fdda2249adb066682e468cfdc6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=220, earliestPutTs=1681034055305 2023-04-09 09:54:16,127 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:16,127 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:16,127 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:16,127 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,141 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:16,141 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:16,146 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): b86d94f6f111f1359f84891af8d12a40#C#compaction#138 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:54:16,146 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,146 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:16,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:16,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:16,147 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:16,147 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:16,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:16,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:16,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:16,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:16,150 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:16,150 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b1240c6[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:16,150 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:16,150 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:16,152 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:16,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:16,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:16,152 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:16,152 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:16,152 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:16,155 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,155 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:16,157 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:16,157 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:16,157 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:16,157 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:16,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:16,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:16,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:16,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:16,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:16,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:16,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:16,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:16,160 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:16,160 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:16,160 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:16,160 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:16,160 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:16,164 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:16,164 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:16,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:16,164 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:16,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:16,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:16,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:16,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:16,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:16,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:16,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:16,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:16,166 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:16,166 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304098f7096f5c48d4f3dab5ffdd892c2f2d7 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304098f7096f5c48d4f3dab5ffdd892c2f2d7 2023-04-09 09:54:16,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:16,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:16,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,168 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:16,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59766 deadline: 1681034116165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,168 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:16,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:16,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,171 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:16,171 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,171 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:16,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:16,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:16,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:16,171 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:16,171 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:16,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:16,171 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:16,171 DEBUG [Thread-888] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:16,171 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:16,171 DEBUG [Thread-888] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:16,171 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,171 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:16,171 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:16,171 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:16,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:16,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:16,174 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:16,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59752 deadline: 1681034116169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,174 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:16,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59822 deadline: 1681034116170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,175 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:16,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59788 deadline: 1681034116170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,175 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:16,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59804 deadline: 1681034116170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,189 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/C/462fc7a5c8ed4c6a9fee53627f78813e as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/462fc7a5c8ed4c6a9fee53627f78813e 2023-04-09 09:54:16,194 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b86d94f6f111f1359f84891af8d12a40/C of b86d94f6f111f1359f84891af8d12a40 into 462fc7a5c8ed4c6a9fee53627f78813e(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:54:16,194 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b86d94f6f111f1359f84891af8d12a40: 2023-04-09 09:54:16,194 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40., storeName=b86d94f6f111f1359f84891af8d12a40/C, priority=12, startTime=1681034056015; duration=0sec 2023-04-09 09:54:16,194 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:54:16,216 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=257, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/A/d5016f012a35410da616ad1c87b7a425 2023-04-09 09:54:16,221 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/A/25712ce6b28640e782c713352b73f1d9 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/25712ce6b28640e782c713352b73f1d9 2023-04-09 09:54:16,232 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b86d94f6f111f1359f84891af8d12a40/A of b86d94f6f111f1359f84891af8d12a40 into 25712ce6b28640e782c713352b73f1d9(size=25.5 K), total size for store is 25.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:54:16,232 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b86d94f6f111f1359f84891af8d12a40: 2023-04-09 09:54:16,232 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40., storeName=b86d94f6f111f1359f84891af8d12a40/A, priority=12, startTime=1681034056014; duration=0sec 2023-04-09 09:54:16,232 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:54:16,241 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=257 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/B/daa806102d3c481490dcca328e605b44 2023-04-09 09:54:16,271 DEBUG [Thread-888] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:16,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:16,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:16,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:16,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:16,277 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:16,277 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:16,277 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:16,277 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:16,277 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:16,277 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:16,279 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,279 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:16,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:16,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:16,279 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,279 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:16,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:16,284 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:16,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:16,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:16,284 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:16,285 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:16,285 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:16,285 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:16,285 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:16,285 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:16,285 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:16,285 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:16,285 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:16,285 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. started... 2023-04-09 09:54:16,285 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. as already flushing 2023-04-09 09:54:16,285 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:16,285 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:16,285 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:16,285 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,285 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:16,285 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:16,287 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,287 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:16,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:16,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:16,287 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:16,287 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:16,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:16,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:16,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:16,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:16,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:16,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@241f2442[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:16,289 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:16,289 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:16,290 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:16,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:16,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:16,290 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:16,290 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:16,290 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:16,291 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,291 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:16,292 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:16,292 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,292 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:16,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:16,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:16,292 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:16,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:16,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:16,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:16,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:16,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:16,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:16,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:16,295 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:16,295 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:16,295 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:16,295 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:16,295 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:16,298 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:16,298 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:16,298 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:16,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:16,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:16,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:16,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:16,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:16,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:16,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:16,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:16,299 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:16,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:16,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:16,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:16,300 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:16,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:16,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,303 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,303 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:16,303 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:16,303 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:16,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:16,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:16,303 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:16,303 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:16,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:16,303 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:16,303 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:16,303 DEBUG [Thread-888] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:16,303 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:16,303 DEBUG [Thread-888] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:16,303 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:16,303 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:16,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:16,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:16,373 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:16,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59766 deadline: 1681034116371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,378 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:16,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59822 deadline: 1681034116375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,379 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:16,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59752 deadline: 1681034116376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,379 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:16,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59788 deadline: 1681034116377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,379 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:16,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59804 deadline: 1681034116377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,404 DEBUG [Thread-888] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:16,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:16,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:16,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:16,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:16,408 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:16,408 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:16,408 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:16,408 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:16,408 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:16,408 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:16,410 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:16,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:16,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:16,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:16,411 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,411 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:16,411 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,411 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:16,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:16,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:16,411 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:16,411 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:16,411 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:16,412 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:16,412 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:16,412 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:16,412 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:16,412 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:16,412 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:16,412 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. started... 2023-04-09 09:54:16,412 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. as already flushing 2023-04-09 09:54:16,412 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:16,413 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:16,413 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:16,413 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,414 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:16,414 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:16,416 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:16,416 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:16,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:16,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:16,416 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:16,416 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:16,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:16,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:16,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:16,418 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:16,418 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:16,418 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b877fef[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:16,418 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:16,420 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:16,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:16,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:16,420 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:16,420 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:16,420 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:16,421 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,421 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:16,422 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:16,422 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:16,422 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:16,422 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:16,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:16,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:16,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:16,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:16,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:16,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:16,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:16,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:16,425 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:16,425 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:16,425 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:16,425 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:16,425 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:16,426 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:16,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:16,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:16,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:16,427 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:16,427 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:16,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:16,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:16,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:16,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:16,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:16,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:16,428 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:16,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:16,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:16,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,429 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:16,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:16,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,434 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,434 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:16,434 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:16,434 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,434 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:16,434 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:16,434 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:16,434 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:16,434 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:16,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:16,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:16,434 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:16,434 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:16,434 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:16,434 DEBUG [Thread-888] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:16,434 DEBUG [Thread-888] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:16,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:16,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:16,534 DEBUG [Thread-888] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:16,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:16,537 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:16,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:16,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:16,538 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:16,538 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:16,538 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:16,538 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:16,539 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:16,539 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:16,541 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:16,541 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:16,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:16,541 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,541 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:16,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:16,541 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:16,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:16,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:16,542 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:16,542 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:16,542 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:16,542 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:16,542 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:16,542 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:16,543 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:16,543 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:16,543 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:16,543 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. started... 2023-04-09 09:54:16,543 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. as already flushing 2023-04-09 09:54:16,543 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:16,543 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:16,543 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:16,543 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,543 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:16,543 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:16,546 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:16,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:16,546 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:16,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:16,546 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:16,546 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:16,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:16,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:16,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:16,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:16,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b0b36b7[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:16,548 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:16,548 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:16,549 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:16,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:16,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:16,549 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:16,549 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:16,549 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:16,549 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,549 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:16,557 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,557 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:16,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:16,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:16,557 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:16,557 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:16,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:16,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:16,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:16,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:16,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:16,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:16,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:16,560 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:16,560 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:16,560 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:16,560 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:16,560 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:16,562 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:16,562 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:16,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:16,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:16,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:16,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:16,562 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:16,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:16,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:16,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:16,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:16,562 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:16,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:16,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:16,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:16,564 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:16,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:16,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,566 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,566 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:16,566 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:16,566 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,566 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:16,566 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:16,566 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:16,566 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:16,566 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:16,566 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:16,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:16,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:16,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:16,566 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:16,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:16,567 DEBUG [Thread-888] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:16,567 DEBUG [Thread-888] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:16,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:16,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:16,656 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=257 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/C/c73c7fdc8e094e2eb4fee221d5f9df93 2023-04-09 09:54:16,667 DEBUG [Thread-888] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:16,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:16,669 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/A/d5016f012a35410da616ad1c87b7a425 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/d5016f012a35410da616ad1c87b7a425 2023-04-09 09:54:16,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:16,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:16,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:16,672 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:16,672 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:16,672 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:16,672 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:16,673 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:16,673 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:16,676 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,676 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/d5016f012a35410da616ad1c87b7a425, entries=250, sequenceid=257, filesize=39.0 K 2023-04-09 09:54:16,676 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,676 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:16,676 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:16,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:16,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:16,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:16,679 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/B/daa806102d3c481490dcca328e605b44 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/daa806102d3c481490dcca328e605b44 2023-04-09 09:54:16,680 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:16,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:16,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:16,681 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:16,692 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:16,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59766 deadline: 1681034116679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,692 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:16,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59822 deadline: 1681034116681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,693 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:16,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59788 deadline: 1681034116682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,693 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:16,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59804 deadline: 1681034116682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,693 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:16,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59752 deadline: 1681034116686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:16,695 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/daa806102d3c481490dcca328e605b44, entries=150, sequenceid=257, filesize=11.8 K 2023-04-09 09:54:16,696 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/C/c73c7fdc8e094e2eb4fee221d5f9df93 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/c73c7fdc8e094e2eb4fee221d5f9df93 2023-04-09 09:54:16,701 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/c73c7fdc8e094e2eb4fee221d5f9df93, entries=150, sequenceid=257, filesize=11.8 K 2023-04-09 09:54:16,702 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:16,702 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:16,702 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:16,702 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:16,702 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:16,702 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=46.96 KB/48090 for b86d94f6f111f1359f84891af8d12a40 in 656ms, sequenceid=257, compaction requested=false 2023-04-09 09:54:16,702 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b86d94f6f111f1359f84891af8d12a40: 2023-04-09 09:54:16,709 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:16,709 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:16,709 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:16,709 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. started... 2023-04-09 09:54:16,710 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b86d94f6f111f1359f84891af8d12a40 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-09 09:54:16,710 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=A 2023-04-09 09:54:16,710 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:16,710 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=B 2023-04-09 09:54:16,710 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:16,710 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=C 2023-04-09 09:54:16,710 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:17,135 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304098a4eb4ea9c0b43aa851e4d1a47104a91 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304098a4eb4ea9c0b43aa851e4d1a47104a91 2023-04-09 09:54:17,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on b86d94f6f111f1359f84891af8d12a40 2023-04-09 09:54:17,197 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. as already flushing 2023-04-09 09:54:17,212 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:17,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59822 deadline: 1681034117210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:17,213 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:17,213 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681034057207, max timestamp=1681034057211Num uniques 859; 2023-04-09 09:54:17,214 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681034057207, max timestamp=1681034057211Num uniques 859; 2023-04-09 09:54:17,214 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-09 09:54:17,214 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681034057207, max timestamp=1681034057211Num uniques 859; 2023-04-09 09:54:17,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59766 deadline: 1681034117211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:17,214 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-09 09:54:17,214 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-09 09:54:17,230 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:17,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59788 deadline: 1681034117228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:17,230 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:17,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59752 deadline: 1681034117229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:17,230 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:17,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59804 deadline: 1681034117229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:17,316 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:17,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59822 deadline: 1681034117315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:17,318 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:17,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59766 deadline: 1681034117315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:17,333 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:17,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59752 deadline: 1681034117331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:17,333 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:17,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59788 deadline: 1681034117331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:17,333 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:17,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59804 deadline: 1681034117332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:17,520 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:17,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59822 deadline: 1681034117517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:17,520 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:17,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59766 deadline: 1681034117519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:17,536 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:17,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59752 deadline: 1681034117534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:17,537 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:17,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59804 deadline: 1681034117535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:17,539 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:17,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59788 deadline: 1681034117535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:17,541 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=270, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/A/982777c39d494d11b27e4f9784f1ab9d 2023-04-09 09:54:17,822 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:17,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59822 deadline: 1681034117822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:17,823 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:17,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59766 deadline: 1681034117823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:17,839 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:17,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59752 deadline: 1681034117838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:17,842 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:17,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59804 deadline: 1681034117840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:17,844 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:17,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59788 deadline: 1681034117842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:17,953 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=270 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/B/fa26293ed3f54ba8932269a959fd7f4b 2023-04-09 09:54:17,984 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=270 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/C/0462cbc291764680b8903cc0b7dd1768 2023-04-09 09:54:17,989 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/A/982777c39d494d11b27e4f9784f1ab9d as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/982777c39d494d11b27e4f9784f1ab9d 2023-04-09 09:54:17,994 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/982777c39d494d11b27e4f9784f1ab9d, entries=150, sequenceid=270, filesize=25.4 K 2023-04-09 09:54:17,994 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/B/fa26293ed3f54ba8932269a959fd7f4b as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/fa26293ed3f54ba8932269a959fd7f4b 2023-04-09 09:54:17,999 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/fa26293ed3f54ba8932269a959fd7f4b, entries=150, sequenceid=270, filesize=11.9 K 2023-04-09 09:54:17,999 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/C/0462cbc291764680b8903cc0b7dd1768 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/0462cbc291764680b8903cc0b7dd1768 2023-04-09 09:54:18,006 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/0462cbc291764680b8903cc0b7dd1768, entries=150, sequenceid=270, filesize=11.9 K 2023-04-09 09:54:18,007 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for b86d94f6f111f1359f84891af8d12a40 in 1298ms, sequenceid=270, compaction requested=true 2023-04-09 09:54:18,007 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b86d94f6f111f1359f84891af8d12a40: 2023-04-09 09:54:18,007 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:18,007 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:18,007 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:18,007 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:18,007 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:18,007 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:18,009 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:18,009 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:18,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:18,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:18,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:18,009 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:18,009 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:18,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:18,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:18,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:18,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:18,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:18,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:18,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22365360[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:18,011 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:18,011 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:18,012 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:18,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:18,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:18,012 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:18,012 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:18,012 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:18,012 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:18,012 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:18,013 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:18,013 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:18,013 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:18,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:18,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:18,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:18,014 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:18,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:18,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:18,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:18,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:18,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:18,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:18,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:18,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:18,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:18,016 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:18,016 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:18,016 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:18,016 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:18,016 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:18,019 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:18,019 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:18,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:18,019 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:18,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:18,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:18,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:18,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:18,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:18,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:18,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:18,020 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:18,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:18,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:18,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:18,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:18,021 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:18,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:18,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:18,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:18,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:18,024 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:18,024 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:18,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:18,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:18,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:18,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:18,025 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:18,025 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:18,025 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:18,025 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:18,025 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:18,025 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:18,025 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:18,026 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:18,026 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:18,026 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:18,026 DEBUG [Thread-888] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:18,026 DEBUG [Thread-888] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:18,029 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:18,127 DEBUG [Thread-888] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:18,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:18,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:18,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:18,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:18,130 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:18,130 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:18,130 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:18,130 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:18,131 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:18,131 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:18,132 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:18,132 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:18,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:18,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:18,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:18,132 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:18,132 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:18,133 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:18,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:18,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:18,133 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:18,133 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:18,133 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:18,133 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:18,134 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:18,134 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:18,140 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:18,140 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:18,140 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:18,140 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. started... 2023-04-09 09:54:18,140 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b86d94f6f111f1359f84891af8d12a40 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-09 09:54:18,140 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=A 2023-04-09 09:54:18,140 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:54:18,140 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=B 2023-04-09 09:54:18,140 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:54:18,140 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=C 2023-04-09 09:54:18,141 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:54:18,186 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230409d3c6457c3d884369b6b32d0b51c29a40 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409d3c6457c3d884369b6b32d0b51c29a40 2023-04-09 09:54:18,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on b86d94f6f111f1359f84891af8d12a40 2023-04-09 09:54:18,329 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. as already flushing 2023-04-09 09:54:18,343 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:18,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59766 deadline: 1681034118338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:18,344 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:18,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59822 deadline: 1681034118339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:18,344 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:18,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59752 deadline: 1681034118344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:18,350 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:18,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59788 deadline: 1681034118347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:18,350 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:18,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59804 deadline: 1681034118348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:18,447 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:18,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59766 deadline: 1681034118445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:18,448 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:18,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59822 deadline: 1681034118445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:18,598 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=306, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/A/1f732d5ef21c462191ca469b42c5f482 2023-04-09 09:54:18,610 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=306 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/B/9442e4d771664dafb9be22d318666831 2023-04-09 09:54:18,651 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:18,651 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:18,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59822 deadline: 1681034118649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:18,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59766 deadline: 1681034118649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:18,953 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:18,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59766 deadline: 1681034118953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:18,955 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:18,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59822 deadline: 1681034118954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,022 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=306 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/C/cae1801e73734fb895fcd50c971832c9 2023-04-09 09:54:19,027 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/A/1f732d5ef21c462191ca469b42c5f482 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/1f732d5ef21c462191ca469b42c5f482 2023-04-09 09:54:19,032 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/1f732d5ef21c462191ca469b42c5f482, entries=150, sequenceid=306, filesize=25.4 K 2023-04-09 09:54:19,033 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/B/9442e4d771664dafb9be22d318666831 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/9442e4d771664dafb9be22d318666831 2023-04-09 09:54:19,037 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/9442e4d771664dafb9be22d318666831, entries=150, sequenceid=306, filesize=11.9 K 2023-04-09 09:54:19,038 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/C/cae1801e73734fb895fcd50c971832c9 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/cae1801e73734fb895fcd50c971832c9 2023-04-09 09:54:19,043 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/cae1801e73734fb895fcd50c971832c9, entries=150, sequenceid=306, filesize=11.9 K 2023-04-09 09:54:19,044 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for b86d94f6f111f1359f84891af8d12a40 in 904ms, sequenceid=306, compaction requested=true 2023-04-09 09:54:19,044 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b86d94f6f111f1359f84891af8d12a40: 2023-04-09 09:54:19,044 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:19,044 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:19,044 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:19,044 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,044 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:19,044 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:19,046 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:19,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:19,046 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:19,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:19,047 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:19,047 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:19,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:19,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:19,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:19,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:19,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f565e50[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:19,048 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:19,048 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:19,049 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:19,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:19,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:19,050 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:19,050 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:19,050 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:19,050 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,050 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:19,052 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,052 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:19,052 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:19,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:19,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:19,052 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:19,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:19,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:19,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:19,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:19,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:19,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:19,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:19,054 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:19,054 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:19,055 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:19,055 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:19,055 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:19,056 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:19,056 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:19,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:19,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:19,056 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:19,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:19,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:19,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:19,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:19,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:19,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:19,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:19,057 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:19,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:19,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:19,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,058 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:19,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:19,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,060 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:19,060 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:19,060 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:19,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:19,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:19,061 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:19,060 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:19,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:19,061 DEBUG [Thread-888] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:19,061 DEBUG [Thread-888] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:19,061 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:19,061 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,061 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:19,061 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:19,061 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:19,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:19,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:19,161 DEBUG [Thread-888] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:19,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:19,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:19,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:19,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:19,166 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:19,166 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:19,166 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:19,166 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:19,166 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:19,166 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:19,167 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,168 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:19,168 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,168 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:19,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:19,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:19,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:19,169 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:19,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:19,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:19,169 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:19,169 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:19,169 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:19,169 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:19,169 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:19,169 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:19,169 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:19,169 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:19,170 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:19,170 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. started... 2023-04-09 09:54:19,170 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b86d94f6f111f1359f84891af8d12a40 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-09 09:54:19,170 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=A 2023-04-09 09:54:19,170 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:19,170 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=B 2023-04-09 09:54:19,170 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:19,170 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=C 2023-04-09 09:54:19,170 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:19,201 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304098b9a3404db8440ce9828194acc60900c to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304098b9a3404db8440ce9828194acc60900c 2023-04-09 09:54:19,206 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=316, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/A/2977c3dba29347f5b7c410a14778065d 2023-04-09 09:54:19,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on b86d94f6f111f1359f84891af8d12a40 2023-04-09 09:54:19,353 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. as already flushing 2023-04-09 09:54:19,378 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:19,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59752 deadline: 1681034119375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,378 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:19,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59804 deadline: 1681034119378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,379 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034059374, max timestamp=1681034059377Num uniques 859; 2023-04-09 09:54:19,379 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034059374, max timestamp=1681034059377Num uniques 859; 2023-04-09 09:54:19,379 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:54:19,379 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:54:19,379 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034059374, max timestamp=1681034059377Num uniques 859; 2023-04-09 09:54:19,379 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:54:19,434 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:19,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59788 deadline: 1681034119432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,458 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:19,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59822 deadline: 1681034119456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,459 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:19,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59766 deadline: 1681034119458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,483 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:19,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59752 deadline: 1681034119479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,483 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:19,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59804 deadline: 1681034119479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,536 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:19,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59788 deadline: 1681034119535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,616 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=316 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/B/cf93cbe385a94d179af0e33422552b1e 2023-04-09 09:54:19,645 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=316 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/C/fad46337be894973995dbbafff399d42 2023-04-09 09:54:19,652 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/A/2977c3dba29347f5b7c410a14778065d as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/2977c3dba29347f5b7c410a14778065d 2023-04-09 09:54:19,658 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/2977c3dba29347f5b7c410a14778065d, entries=150, sequenceid=316, filesize=25.4 K 2023-04-09 09:54:19,659 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/B/cf93cbe385a94d179af0e33422552b1e as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/cf93cbe385a94d179af0e33422552b1e 2023-04-09 09:54:19,669 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/cf93cbe385a94d179af0e33422552b1e, entries=150, sequenceid=316, filesize=11.9 K 2023-04-09 09:54:19,670 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/C/fad46337be894973995dbbafff399d42 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/fad46337be894973995dbbafff399d42 2023-04-09 09:54:19,676 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/fad46337be894973995dbbafff399d42, entries=150, sequenceid=316, filesize=11.9 K 2023-04-09 09:54:19,677 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for b86d94f6f111f1359f84891af8d12a40 in 507ms, sequenceid=316, compaction requested=true 2023-04-09 09:54:19,677 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b86d94f6f111f1359f84891af8d12a40: 2023-04-09 09:54:19,677 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:19,677 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:19,677 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:19,677 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,677 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:19,677 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:19,679 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:19,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:19,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:19,680 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:19,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:19,680 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:19,680 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:19,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:19,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:19,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:19,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@fde2e52[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:19,681 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:19,681 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:19,684 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:19,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:19,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:19,684 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:19,684 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:19,684 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:19,684 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,684 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:19,685 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:19,685 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:19,685 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,685 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:19,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:19,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:19,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:19,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:19,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:19,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:19,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:19,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on b86d94f6f111f1359f84891af8d12a40 2023-04-09 09:54:19,687 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b86d94f6f111f1359f84891af8d12a40 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-04-09 09:54:19,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,689 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681034059409, max timestamp=1681034059686Num uniques 1018; 2023-04-09 09:54:19,689 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-09 09:54:19,689 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=A 2023-04-09 09:54:19,689 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-09 09:54:19,689 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=B 2023-04-09 09:54:19,689 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:54:19,689 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=C 2023-04-09 09:54:19,689 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-09 09:54:19,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:19,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:19,691 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:19,692 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:19,692 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:19,692 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:19,692 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:19,698 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:19,698 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:19,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:19,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:19,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:19,698 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:19,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:19,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:19,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:19,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:19,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:19,699 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:19,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:19,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:19,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:19,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,702 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:19,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:19,705 WARN [DataStreamer for file /user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230409c8221449e9264009a11d122a6ee72efe] hdfs.DataStreamer(982): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) at java.lang.Thread.join(Thread.java:1331) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:630) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:807) 2023-04-09 09:54:19,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,713 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,713 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:19,713 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:19,713 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,713 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:19,714 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:19,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:19,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:19,713 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:19,714 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:19,714 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:19,714 DEBUG [Thread-888] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:19,714 DEBUG [Thread-888] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:19,714 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:19,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:19,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:19,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:19,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:19,716 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230409c8221449e9264009a11d122a6ee72efe to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409c8221449e9264009a11d122a6ee72efe 2023-04-09 09:54:19,721 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:19,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59752 deadline: 1681034119717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,722 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:19,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59804 deadline: 1681034119719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,738 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=355, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/A/d876dbb86989410a839bfd16fb7c24e3 2023-04-09 09:54:19,742 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:19,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59788 deadline: 1681034119738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,754 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=355 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/B/1ccaa286287040e9aab6fc56326b943a 2023-04-09 09:54:19,814 DEBUG [Thread-888] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:19,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:19,816 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:19,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:19,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:19,818 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:19,818 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:19,818 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:19,818 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:19,818 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:19,818 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:19,819 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,819 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:19,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:19,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:19,820 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,820 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:19,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:19,820 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:19,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:19,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:19,821 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:19,821 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:19,821 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:19,821 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:19,821 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:19,821 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:19,821 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:19,821 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:19,821 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:19,821 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. started... 2023-04-09 09:54:19,821 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. as already flushing 2023-04-09 09:54:19,821 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:19,821 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:19,821 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:19,821 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,821 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:19,822 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:19,823 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,823 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:19,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:19,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:19,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:19,824 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:19,824 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:19,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:19,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:19,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:19,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:19,826 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:19,826 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:19,826 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d320f90[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:19,826 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:19,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59752 deadline: 1681034119824, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,826 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:19,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59804 deadline: 1681034119825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,828 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:19,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:19,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:19,828 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:19,828 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:19,828 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:19,828 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,828 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:19,829 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,829 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:19,829 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:19,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:19,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:19,829 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:19,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:19,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:19,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:19,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:19,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:19,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:19,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:19,832 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:19,832 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:19,832 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:19,832 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:19,832 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:19,833 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:19,833 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:19,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:19,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:19,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:19,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:19,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:19,834 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:19,834 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:19,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:19,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:19,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:19,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:19,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:19,835 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:19,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:19,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:19,837 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,837 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:19,837 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:19,837 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,837 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:19,837 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:19,837 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:19,838 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:19,838 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:19,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:19,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:19,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:19,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:19,838 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:19,838 DEBUG [Thread-888] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:19,838 DEBUG [Thread-888] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:19,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:19,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:19,838 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:19,938 DEBUG [Thread-888] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:19,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:19,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:19,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:19,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:19,942 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:19,942 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:19,942 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:19,942 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:19,942 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:19,942 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:19,944 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,944 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:19,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:19,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:19,944 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,944 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:19,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:19,944 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:19,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:19,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:19,945 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:19,945 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:19,945 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:19,945 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:19,945 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:19,945 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:19,945 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:19,945 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:19,945 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:19,945 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. started... 2023-04-09 09:54:19,945 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. as already flushing 2023-04-09 09:54:19,946 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:19,946 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:19,946 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:19,946 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,946 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:19,946 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:19,947 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,947 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:19,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:19,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:19,947 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:19,947 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:19,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:19,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:19,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:19,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:19,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:19,949 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:19,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d0fea05[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:19,949 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:19,950 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:19,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:19,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:19,950 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:19,950 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:19,950 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:19,950 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,950 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:19,952 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,952 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:19,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:19,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:19,953 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:19,953 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:19,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:19,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:19,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:19,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:19,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:19,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:19,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:19,955 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:19,955 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:19,955 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:19,955 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:19,955 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:19,956 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:19,956 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:19,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:19,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:19,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:19,957 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:19,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:19,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:19,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:19,957 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:19,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:19,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:19,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:19,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:19,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:19,958 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:19,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:19,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,961 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,961 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:19,961 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:19,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:19,961 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:19,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:19,961 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:19,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:19,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:19,961 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:19,961 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:19,961 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:19,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:19,961 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:19,961 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:19,961 DEBUG [Thread-888] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:19,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:19,961 DEBUG [Thread-888] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:19,961 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:20,029 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:20,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59804 deadline: 1681034120027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:20,029 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:20,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59752 deadline: 1681034120027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:20,048 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:20,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59788 deadline: 1681034120047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:20,062 DEBUG [Thread-888] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:20,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:20,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:20,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:20,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:20,065 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:20,065 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:20,065 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:20,065 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:20,065 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:20,066 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:20,067 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:20,067 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:20,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:20,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:20,067 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:20,067 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:20,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:20,067 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:20,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:20,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:20,068 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:20,068 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:20,068 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:20,068 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:20,068 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:20,069 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:20,069 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:20,069 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:20,069 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:20,069 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. started... 2023-04-09 09:54:20,069 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. as already flushing 2023-04-09 09:54:20,069 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:20,069 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:20,069 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:20,069 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:20,069 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:20,069 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:20,072 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:20,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:20,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:20,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:20,072 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:20,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:20,072 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:20,072 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:20,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:20,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:20,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:20,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:20,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:20,074 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:20,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66992a[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:20,074 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:20,075 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:20,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:20,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:20,075 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:20,075 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:20,075 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:20,075 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:20,075 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:20,076 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:20,076 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:20,076 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:20,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:20,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:20,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:20,076 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:20,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:20,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:20,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:20,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:20,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:20,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:20,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:20,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:20,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:20,079 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:20,079 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:20,079 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:20,079 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:20,079 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:20,080 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:20,080 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:20,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:20,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:20,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:20,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:20,080 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:20,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:20,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:20,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:20,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:20,081 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:20,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:20,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:20,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:20,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:20,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:20,082 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:20,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:20,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:20,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:20,087 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:20,087 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:20,087 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:20,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:20,087 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:20,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:20,087 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:20,087 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:20,087 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:20,087 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:20,087 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:20,087 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:20,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:20,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:20,087 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:20,088 DEBUG [Thread-888] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:20,088 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:20,088 DEBUG [Thread-888] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:20,165 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=355 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/C/f1ff3b5bae234e8cbffbae27591223a5 2023-04-09 09:54:20,170 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/A/d876dbb86989410a839bfd16fb7c24e3 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/d876dbb86989410a839bfd16fb7c24e3 2023-04-09 09:54:20,175 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/d876dbb86989410a839bfd16fb7c24e3, entries=250, sequenceid=355, filesize=39.2 K 2023-04-09 09:54:20,176 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/B/1ccaa286287040e9aab6fc56326b943a as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/1ccaa286287040e9aab6fc56326b943a 2023-04-09 09:54:20,180 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/1ccaa286287040e9aab6fc56326b943a, entries=150, sequenceid=355, filesize=11.9 K 2023-04-09 09:54:20,181 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/C/f1ff3b5bae234e8cbffbae27591223a5 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/f1ff3b5bae234e8cbffbae27591223a5 2023-04-09 09:54:20,185 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/f1ff3b5bae234e8cbffbae27591223a5, entries=150, sequenceid=355, filesize=11.9 K 2023-04-09 09:54:20,186 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~109.58 KB/112210, heapSize ~312.08 KB/319568, currentSize=67.09 KB/68700 for b86d94f6f111f1359f84891af8d12a40 in 499ms, sequenceid=355, compaction requested=true 2023-04-09 09:54:20,186 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b86d94f6f111f1359f84891af8d12a40: 2023-04-09 09:54:20,186 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:54:20,186 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-09 09:54:20,186 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-09 09:54:20,186 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:54:20,186 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:54:20,188 DEBUG [Thread-888] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:20,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:20,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:20,190 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 184334 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-09 09:54:20,191 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1912): b86d94f6f111f1359f84891af8d12a40/A is initiating minor compaction (all files) 2023-04-09 09:54:20,191 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b86d94f6f111f1359f84891af8d12a40/A in TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:20,191 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73106 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-09 09:54:20,191 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/25712ce6b28640e782c713352b73f1d9, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/d5016f012a35410da616ad1c87b7a425, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/982777c39d494d11b27e4f9784f1ab9d, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/1f732d5ef21c462191ca469b42c5f482, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/2977c3dba29347f5b7c410a14778065d, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/d876dbb86989410a839bfd16fb7c24e3] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp, totalSize=180.0 K 2023-04-09 09:54:20,191 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): b86d94f6f111f1359f84891af8d12a40/B is initiating minor compaction (all files) 2023-04-09 09:54:20,191 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b86d94f6f111f1359f84891af8d12a40/B in TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:20,191 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/25adbe40e82e4215a301c73de4368b9f, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/daa806102d3c481490dcca328e605b44, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/fa26293ed3f54ba8932269a959fd7f4b, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/9442e4d771664dafb9be22d318666831, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/cf93cbe385a94d179af0e33422552b1e, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/1ccaa286287040e9aab6fc56326b943a] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp, totalSize=71.4 K 2023-04-09 09:54:20,191 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 25712ce6b28640e782c713352b73f1d9, keycount=150, bloomtype=ROW, size=25.5 K, encoding=NONE, compression=NONE, seqNum=220, earliestPutTs=1681034055305 2023-04-09 09:54:20,196 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 25adbe40e82e4215a301c73de4368b9f, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=220, earliestPutTs=1681034055305 2023-04-09 09:54:20,196 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting d5016f012a35410da616ad1c87b7a425, keycount=250, bloomtype=ROW, size=39.0 K, encoding=NONE, compression=NONE, seqNum=257, earliestPutTs=1681034055929 2023-04-09 09:54:20,197 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting daa806102d3c481490dcca328e605b44, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=257, earliestPutTs=1681034055937 2023-04-09 09:54:20,197 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 982777c39d494d11b27e4f9784f1ab9d, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=270, earliestPutTs=1681034056056 2023-04-09 09:54:20,197 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting fa26293ed3f54ba8932269a959fd7f4b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=270, earliestPutTs=1681034056056 2023-04-09 09:54:20,197 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 1f732d5ef21c462191ca469b42c5f482, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=306, earliestPutTs=1681034057218 2023-04-09 09:54:20,197 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 9442e4d771664dafb9be22d318666831, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=306, earliestPutTs=1681034057218 2023-04-09 09:54:20,198 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 2977c3dba29347f5b7c410a14778065d, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=316, earliestPutTs=1681034058332 2023-04-09 09:54:20,198 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting cf93cbe385a94d179af0e33422552b1e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=316, earliestPutTs=1681034058332 2023-04-09 09:54:20,198 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting d876dbb86989410a839bfd16fb7c24e3, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=355, earliestPutTs=1681034059409 2023-04-09 09:54:20,198 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 1ccaa286287040e9aab6fc56326b943a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=355, earliestPutTs=1681034059416 2023-04-09 09:54:20,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:20,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:20,208 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:20,208 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:20,208 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:20,209 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:20,209 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:20,209 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:20,212 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:20,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:20,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:20,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:20,213 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:20,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:20,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:20,213 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:20,213 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:20,213 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:20,213 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:20,213 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): b86d94f6f111f1359f84891af8d12a40#B#compaction#154 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:54:20,213 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:20,214 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:20,214 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:20,215 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:20,215 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:20,216 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:20,216 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:20,216 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:20,216 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. started... 2023-04-09 09:54:20,216 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b86d94f6f111f1359f84891af8d12a40 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-09 09:54:20,216 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=A 2023-04-09 09:54:20,216 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:20,216 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=B 2023-04-09 09:54:20,216 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:20,216 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=C 2023-04-09 09:54:20,217 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:20,219 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] throttle.PressureAwareThroughputController(145): b86d94f6f111f1359f84891af8d12a40#A#compaction#153 average throughput is 1.97 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:54:20,230 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/B/066e6b517f3e44cbbb7a82804d7de002 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/066e6b517f3e44cbbb7a82804d7de002 2023-04-09 09:54:20,237 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in b86d94f6f111f1359f84891af8d12a40/B of b86d94f6f111f1359f84891af8d12a40 into 066e6b517f3e44cbbb7a82804d7de002(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:54:20,237 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b86d94f6f111f1359f84891af8d12a40: 2023-04-09 09:54:20,237 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40., storeName=b86d94f6f111f1359f84891af8d12a40/B, priority=10, startTime=1681034060186; duration=0sec 2023-04-09 09:54:20,237 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:54:20,237 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-09 09:54:20,239 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73106 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-09 09:54:20,239 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): b86d94f6f111f1359f84891af8d12a40/C is initiating minor compaction (all files) 2023-04-09 09:54:20,240 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b86d94f6f111f1359f84891af8d12a40/C in TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:20,240 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/462fc7a5c8ed4c6a9fee53627f78813e, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/c73c7fdc8e094e2eb4fee221d5f9df93, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/0462cbc291764680b8903cc0b7dd1768, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/cae1801e73734fb895fcd50c971832c9, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/fad46337be894973995dbbafff399d42, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/f1ff3b5bae234e8cbffbae27591223a5] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp, totalSize=71.4 K 2023-04-09 09:54:20,240 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 462fc7a5c8ed4c6a9fee53627f78813e, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=220, earliestPutTs=1681034055305 2023-04-09 09:54:20,241 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting c73c7fdc8e094e2eb4fee221d5f9df93, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=257, earliestPutTs=1681034055937 2023-04-09 09:54:20,242 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 0462cbc291764680b8903cc0b7dd1768, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=270, earliestPutTs=1681034056056 2023-04-09 09:54:20,242 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting cae1801e73734fb895fcd50c971832c9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=306, earliestPutTs=1681034057218 2023-04-09 09:54:20,243 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting fad46337be894973995dbbafff399d42, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=316, earliestPutTs=1681034058332 2023-04-09 09:54:20,243 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting f1ff3b5bae234e8cbffbae27591223a5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=355, earliestPutTs=1681034059416 2023-04-09 09:54:20,253 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): b86d94f6f111f1359f84891af8d12a40#C#compaction#156 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:54:20,264 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/C/5effa40e1d1d4c9bb00783c2b6211c39 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/5effa40e1d1d4c9bb00783c2b6211c39 2023-04-09 09:54:20,268 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in b86d94f6f111f1359f84891af8d12a40/C of b86d94f6f111f1359f84891af8d12a40 into 5effa40e1d1d4c9bb00783c2b6211c39(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:54:20,269 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b86d94f6f111f1359f84891af8d12a40: 2023-04-09 09:54:20,269 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40., storeName=b86d94f6f111f1359f84891af8d12a40/C, priority=10, startTime=1681034060186; duration=0sec 2023-04-09 09:54:20,269 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:54:20,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on b86d94f6f111f1359f84891af8d12a40 2023-04-09 09:54:20,332 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. as already flushing 2023-04-09 09:54:20,365 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:20,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59804 deadline: 1681034120363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:20,366 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:20,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59752 deadline: 1681034120364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:20,465 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:20,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59822 deadline: 1681034120464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:20,465 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:20,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59766 deadline: 1681034120464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:20,469 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:20,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59804 deadline: 1681034120466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:20,469 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:20,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59752 deadline: 1681034120466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:20,552 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:20,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59788 deadline: 1681034120549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:20,650 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230409c738f228fb58424c8cd3317dcb65ecfc to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409c738f228fb58424c8cd3317dcb65ecfc 2023-04-09 09:54:20,651 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/A/5cc3a69de617432e81080d9f250c12c5 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/5cc3a69de617432e81080d9f250c12c5 2023-04-09 09:54:20,659 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in b86d94f6f111f1359f84891af8d12a40/A of b86d94f6f111f1359f84891af8d12a40 into 5cc3a69de617432e81080d9f250c12c5(size=25.9 K), total size for store is 25.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:54:20,659 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b86d94f6f111f1359f84891af8d12a40: 2023-04-09 09:54:20,660 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40., storeName=b86d94f6f111f1359f84891af8d12a40/A, priority=10, startTime=1681034060186; duration=0sec 2023-04-09 09:54:20,660 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:54:20,670 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:20,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59804 deadline: 1681034120670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:20,674 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:20,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59752 deadline: 1681034120671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:20,972 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:20,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59804 deadline: 1681034120972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:20,977 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:20,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59752 deadline: 1681034120975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:21,060 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=368, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/A/7330f05ddf164e649d24a063e1bb5fe3 2023-04-09 09:54:21,075 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=368 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/B/2936bbce478643c8a70605ac0b915e2a 2023-04-09 09:54:21,475 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:21,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59804 deadline: 1681034121475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:21,482 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:21,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59752 deadline: 1681034121482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:21,488 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=368 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/C/f524438947784a3aadec83f415494a36 2023-04-09 09:54:21,493 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/A/7330f05ddf164e649d24a063e1bb5fe3 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/7330f05ddf164e649d24a063e1bb5fe3 2023-04-09 09:54:21,497 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/7330f05ddf164e649d24a063e1bb5fe3, entries=150, sequenceid=368, filesize=25.4 K 2023-04-09 09:54:21,498 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/B/2936bbce478643c8a70605ac0b915e2a as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/2936bbce478643c8a70605ac0b915e2a 2023-04-09 09:54:21,502 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/2936bbce478643c8a70605ac0b915e2a, entries=150, sequenceid=368, filesize=11.9 K 2023-04-09 09:54:21,503 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/C/f524438947784a3aadec83f415494a36 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/f524438947784a3aadec83f415494a36 2023-04-09 09:54:21,508 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/f524438947784a3aadec83f415494a36, entries=150, sequenceid=368, filesize=11.9 K 2023-04-09 09:54:21,509 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for b86d94f6f111f1359f84891af8d12a40 in 1293ms, sequenceid=368, compaction requested=false 2023-04-09 09:54:21,509 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b86d94f6f111f1359f84891af8d12a40: 2023-04-09 09:54:21,509 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:21,509 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:21,509 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:21,509 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:21,514 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:21,514 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:21,517 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:21,517 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:21,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:21,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:21,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:21,517 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:21,517 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:21,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:21,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:21,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:21,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:21,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:21,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:21,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7eb49bca[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:21,519 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:21,519 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:21,520 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:21,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:21,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:21,520 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:21,520 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:21,520 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:21,520 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:21,520 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:21,521 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:21,521 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:21,521 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:21,521 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:21,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:21,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:21,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:21,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:21,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:21,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:21,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:21,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:21,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:21,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:21,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:21,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:21,524 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:21,524 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:21,524 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:21,524 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:21,524 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:21,525 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:21,525 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:21,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:21,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:21,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:21,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:21,526 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:21,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:21,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:21,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:21,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:21,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:21,526 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:21,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:21,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:21,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:21,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:21,528 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:21,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:21,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:21,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:21,531 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:21,532 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:21,532 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:21,532 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:21,532 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:21,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:21,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:21,532 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:21,532 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:21,532 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:21,532 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:21,532 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:21,532 DEBUG [Thread-888] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:21,532 DEBUG [Thread-888] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:21,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:21,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:21,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:21,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:21,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on b86d94f6f111f1359f84891af8d12a40 2023-04-09 09:54:21,561 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b86d94f6f111f1359f84891af8d12a40 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-09 09:54:21,561 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=A 2023-04-09 09:54:21,561 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:21,561 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=B 2023-04-09 09:54:21,561 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:21,561 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=C 2023-04-09 09:54:21,562 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:21,606 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:21,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59788 deadline: 1681034121605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:21,611 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304097fb5ddb6969f4f0c847e04be9d5a85cd to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304097fb5ddb6969f4f0c847e04be9d5a85cd 2023-04-09 09:54:21,621 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=391, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/A/a85aea19f47a4ac3ba66ea4e58a01967 2023-04-09 09:54:21,632 DEBUG [Thread-888] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:21,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:21,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:21,637 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=391 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/B/a6b7199945d24ebf959dc32ba4d559eb 2023-04-09 09:54:21,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:21,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:21,637 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:21,637 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:21,638 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:21,638 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:21,638 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:21,638 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:21,639 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:21,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:21,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:21,640 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:21,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:21,641 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:21,641 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:21,641 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:21,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:21,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:21,642 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:21,642 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:21,642 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:21,642 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:21,642 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:21,643 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:21,643 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:21,643 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:21,643 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. started... 2023-04-09 09:54:21,643 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. as already flushing 2023-04-09 09:54:21,643 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:21,643 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:21,643 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:21,643 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:21,643 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:21,643 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:21,645 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:21,645 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:21,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:21,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:21,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:21,645 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:21,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:21,645 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:21,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:21,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:21,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:21,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:21,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:21,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33b74503[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:21,647 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:21,647 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:21,648 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:21,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:21,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:21,648 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:21,648 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:21,648 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:21,648 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:21,648 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:21,650 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:21,650 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:21,650 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:21,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:21,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:21,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:21,650 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:21,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:21,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:21,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:21,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:21,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:21,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:21,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:21,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:21,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:21,653 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:21,653 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:21,653 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:21,653 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:21,653 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:21,654 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:21,654 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:21,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:21,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:21,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:21,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:21,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:21,654 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:21,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:21,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:21,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:21,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:21,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:21,655 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:21,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:21,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:21,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:21,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:21,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:21,658 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:21,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:21,661 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:21,662 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:21,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:21,662 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:21,662 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:21,662 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:21,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:21,662 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:21,662 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:21,662 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:21,662 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:21,662 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:21,662 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:21,662 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:21,662 DEBUG [Thread-888] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:21,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:21,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:21,662 DEBUG [Thread-888] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:21,709 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:21,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59788 deadline: 1681034121707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:21,762 DEBUG [Thread-888] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:21,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:21,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:21,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:21,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:21,766 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:21,766 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:21,766 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:21,766 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:21,767 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:21,767 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:21,769 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:21,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:21,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:21,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:21,769 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:21,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:21,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:21,770 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:21,771 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:21,771 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:21,771 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:21,771 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:21,771 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:21,771 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:21,771 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:21,771 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:21,771 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. started... 2023-04-09 09:54:21,771 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:21,771 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. as already flushing 2023-04-09 09:54:21,771 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:21,771 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:21,771 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:21,771 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:21,772 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:21,772 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:21,772 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:21,772 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:21,773 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:21,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:21,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:21,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:21,773 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:21,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:21,774 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:21,774 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:21,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:21,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:21,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:21,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:21,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:21,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1daceb2[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:21,776 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:21,776 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:21,777 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:21,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:21,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:21,777 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:21,777 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:21,777 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:21,777 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:21,777 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:21,780 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:21,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:21,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:21,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:21,785 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:21,785 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:21,785 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:21,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:21,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:21,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:21,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:21,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:21,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:21,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:21,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:21,789 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:21,789 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:21,789 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:21,789 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:21,789 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:21,789 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:21,791 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:21,791 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:21,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:21,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:21,791 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:21,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:21,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:21,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:21,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:21,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:21,791 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:21,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:21,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:21,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:21,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:21,808 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:21,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:21,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:21,812 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:21,813 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:21,813 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:21,813 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:21,813 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:21,813 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:21,813 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:21,813 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:21,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:21,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:21,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:21,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:21,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:21,813 DEBUG [Thread-888] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:21,813 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:21,814 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:21,813 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:21,813 DEBUG [Thread-888] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:21,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:21,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:21,913 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:21,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59788 deadline: 1681034121911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:21,914 DEBUG [Thread-888] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:21,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:21,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:21,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:21,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:21,918 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:21,918 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:21,918 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:21,918 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:21,918 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:21,918 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:21,921 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:21,921 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:21,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:21,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:21,921 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:21,921 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:21,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:21,922 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:21,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:21,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:21,922 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:21,922 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:21,922 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:21,922 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:21,922 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:21,922 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:21,923 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:21,923 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:21,923 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:21,923 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. started... 2023-04-09 09:54:21,923 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. as already flushing 2023-04-09 09:54:21,923 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:21,923 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:21,923 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:21,923 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:21,923 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:21,923 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:21,924 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:21,924 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:21,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:21,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:21,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:21,925 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:21,925 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:21,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:21,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:21,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:21,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:21,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:21,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:21,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6667d073[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:21,926 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:21,926 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:21,927 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:21,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:21,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:21,927 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:21,927 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:21,927 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:21,928 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:21,928 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:21,929 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:21,929 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:21,929 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:21,929 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:21,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:21,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:21,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:21,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:21,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:21,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:21,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:21,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:21,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:21,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:21,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:21,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:21,931 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:21,931 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:21,931 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:21,931 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:21,931 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:21,934 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:21,934 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:21,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:21,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:21,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:21,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:21,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:21,934 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:21,934 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:21,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:21,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:21,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:21,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:21,936 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:21,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:21,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:21,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:21,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:21,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:21,938 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:21,938 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:21,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:21,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:21,938 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:21,938 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:21,938 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:21,938 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:21,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:21,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:21,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:21,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:21,938 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:21,938 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:21,938 DEBUG [Thread-888] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:21,938 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:21,938 DEBUG [Thread-888] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:21,938 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:21,938 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:22,039 DEBUG [Thread-888] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:22,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:22,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:22,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:22,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:22,042 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:22,042 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:22,042 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:22,042 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:22,043 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:22,043 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:22,044 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:22,044 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:22,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:22,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:22,044 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:22,044 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:22,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:22,045 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:22,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:22,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:22,045 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:22,045 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:22,045 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:22,045 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:22,045 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:22,045 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:22,045 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:22,046 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:22,046 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:22,046 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. started... 2023-04-09 09:54:22,046 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. as already flushing 2023-04-09 09:54:22,046 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:22,046 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:22,046 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:22,046 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:22,046 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:22,046 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:22,047 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:22,047 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:22,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:22,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:22,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:22,048 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:22,048 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:22,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:22,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:22,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:22,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:22,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:22,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:22,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@319f2d53[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:22,049 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:22,049 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:22,051 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:22,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:22,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:22,052 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:22,052 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:22,052 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:22,052 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:22,052 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:22,053 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:22,053 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:22,053 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:22,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:22,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:22,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:22,053 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:22,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:22,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:22,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:22,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:22,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:22,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:22,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:22,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:22,056 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:22,056 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:22,056 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:22,056 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:22,056 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:22,056 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:22,057 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:22,057 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:22,057 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:22,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:22,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:22,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:22,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:22,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:22,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:22,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:22,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:22,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:22,059 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:22,061 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=391 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/C/a02c593eed0a423990ea56399773ece8 2023-04-09 09:54:22,066 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/A/a85aea19f47a4ac3ba66ea4e58a01967 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/a85aea19f47a4ac3ba66ea4e58a01967 2023-04-09 09:54:22,071 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/a85aea19f47a4ac3ba66ea4e58a01967, entries=150, sequenceid=391, filesize=25.4 K 2023-04-09 09:54:22,072 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/B/a6b7199945d24ebf959dc32ba4d559eb as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/a6b7199945d24ebf959dc32ba4d559eb 2023-04-09 09:54:22,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:22,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:22,073 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:22,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:22,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:22,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:22,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:22,077 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/a6b7199945d24ebf959dc32ba4d559eb, entries=150, sequenceid=391, filesize=11.9 K 2023-04-09 09:54:22,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:22,078 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/C/a02c593eed0a423990ea56399773ece8 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/a02c593eed0a423990ea56399773ece8 2023-04-09 09:54:22,078 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:22,078 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:22,079 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:22,079 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:22,079 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:22,078 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:22,079 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:22,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:22,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:22,079 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:22,079 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:22,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:22,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:22,079 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:22,079 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:22,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:22,079 DEBUG [Thread-888] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:22,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:22,079 DEBUG [Thread-888] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:22,084 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/a02c593eed0a423990ea56399773ece8, entries=150, sequenceid=391, filesize=11.9 K 2023-04-09 09:54:22,085 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for b86d94f6f111f1359f84891af8d12a40 in 524ms, sequenceid=391, compaction requested=true 2023-04-09 09:54:22,085 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b86d94f6f111f1359f84891af8d12a40: 2023-04-09 09:54:22,085 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:54:22,085 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-09 09:54:22,085 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-09 09:54:22,085 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:54:22,085 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:54:22,086 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 78582 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-09 09:54:22,087 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1912): b86d94f6f111f1359f84891af8d12a40/A is initiating minor compaction (all files) 2023-04-09 09:54:22,087 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b86d94f6f111f1359f84891af8d12a40/A in TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:22,087 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/5cc3a69de617432e81080d9f250c12c5, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/7330f05ddf164e649d24a063e1bb5fe3, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/a85aea19f47a4ac3ba66ea4e58a01967] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp, totalSize=76.7 K 2023-04-09 09:54:22,087 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37002 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-09 09:54:22,087 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): b86d94f6f111f1359f84891af8d12a40/B is initiating minor compaction (all files) 2023-04-09 09:54:22,087 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b86d94f6f111f1359f84891af8d12a40/B in TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:22,087 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/066e6b517f3e44cbbb7a82804d7de002, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/2936bbce478643c8a70605ac0b915e2a, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/a6b7199945d24ebf959dc32ba4d559eb] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp, totalSize=36.1 K 2023-04-09 09:54:22,087 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 5cc3a69de617432e81080d9f250c12c5, keycount=150, bloomtype=ROW, size=25.9 K, encoding=NONE, compression=NONE, seqNum=355, earliestPutTs=1681034059416 2023-04-09 09:54:22,088 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 066e6b517f3e44cbbb7a82804d7de002, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=355, earliestPutTs=1681034059416 2023-04-09 09:54:22,088 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 7330f05ddf164e649d24a063e1bb5fe3, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=368, earliestPutTs=1681034059711 2023-04-09 09:54:22,089 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 2936bbce478643c8a70605ac0b915e2a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=368, earliestPutTs=1681034059711 2023-04-09 09:54:22,089 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting a85aea19f47a4ac3ba66ea4e58a01967, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=391, earliestPutTs=1681034060356 2023-04-09 09:54:22,089 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting a6b7199945d24ebf959dc32ba4d559eb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=391, earliestPutTs=1681034060356 2023-04-09 09:54:22,097 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): b86d94f6f111f1359f84891af8d12a40#B#compaction#162 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:54:22,116 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] throttle.PressureAwareThroughputController(145): b86d94f6f111f1359f84891af8d12a40#A#compaction#163 average throughput is 1.10 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:54:22,121 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/B/0dcd174fd5394f7eb7fe53a42910ce19 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/0dcd174fd5394f7eb7fe53a42910ce19 2023-04-09 09:54:22,127 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b86d94f6f111f1359f84891af8d12a40/B of b86d94f6f111f1359f84891af8d12a40 into 0dcd174fd5394f7eb7fe53a42910ce19(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:54:22,127 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b86d94f6f111f1359f84891af8d12a40: 2023-04-09 09:54:22,127 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40., storeName=b86d94f6f111f1359f84891af8d12a40/B, priority=13, startTime=1681034062085; duration=0sec 2023-04-09 09:54:22,127 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:54:22,127 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-09 09:54:22,129 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37002 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-09 09:54:22,129 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): b86d94f6f111f1359f84891af8d12a40/C is initiating minor compaction (all files) 2023-04-09 09:54:22,129 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b86d94f6f111f1359f84891af8d12a40/C in TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:22,129 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/5effa40e1d1d4c9bb00783c2b6211c39, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/f524438947784a3aadec83f415494a36, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/a02c593eed0a423990ea56399773ece8] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp, totalSize=36.1 K 2023-04-09 09:54:22,129 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 5effa40e1d1d4c9bb00783c2b6211c39, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=355, earliestPutTs=1681034059416 2023-04-09 09:54:22,130 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting f524438947784a3aadec83f415494a36, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=368, earliestPutTs=1681034059711 2023-04-09 09:54:22,130 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting a02c593eed0a423990ea56399773ece8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=391, earliestPutTs=1681034060356 2023-04-09 09:54:22,137 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): b86d94f6f111f1359f84891af8d12a40#C#compaction#164 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:54:22,155 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/C/e5f9f351093d47979a26310cec35f896 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/e5f9f351093d47979a26310cec35f896 2023-04-09 09:54:22,160 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b86d94f6f111f1359f84891af8d12a40/C of b86d94f6f111f1359f84891af8d12a40 into e5f9f351093d47979a26310cec35f896(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:54:22,160 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b86d94f6f111f1359f84891af8d12a40: 2023-04-09 09:54:22,160 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40., storeName=b86d94f6f111f1359f84891af8d12a40/C, priority=13, startTime=1681034062085; duration=0sec 2023-04-09 09:54:22,160 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:54:22,179 DEBUG [Thread-888] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:22,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:22,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:22,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:22,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:22,183 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:22,183 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:22,183 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:22,183 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:22,183 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:22,183 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:22,185 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:22,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:22,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:22,185 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:22,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:22,186 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:22,186 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:22,186 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:22,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:22,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:22,187 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:22,187 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:22,187 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:22,187 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:22,187 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:22,187 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:22,187 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:22,187 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:22,187 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:22,187 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. started... 2023-04-09 09:54:22,187 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b86d94f6f111f1359f84891af8d12a40 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-09 09:54:22,188 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=A 2023-04-09 09:54:22,188 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:22,188 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=B 2023-04-09 09:54:22,188 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:22,188 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=C 2023-04-09 09:54:22,188 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:22,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on b86d94f6f111f1359f84891af8d12a40 2023-04-09 09:54:22,216 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. as already flushing 2023-04-09 09:54:22,221 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230409dfde11793f3a4636952443c3860d3e96 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409dfde11793f3a4636952443c3860d3e96 2023-04-09 09:54:22,228 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=405, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/A/deec1cf53c2c478d849a356122f3ee70 2023-04-09 09:54:22,260 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:22,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59788 deadline: 1681034122258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:22,363 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:22,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59788 deadline: 1681034122361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:22,469 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:22,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59822 deadline: 1681034122468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:22,471 DEBUG [Thread-878] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4131 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40., hostname=jenkins-hbase4.apache.org,42467,1681034019544, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-09 09:54:22,474 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:22,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59766 deadline: 1681034122473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:22,475 DEBUG [Thread-880] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4136 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40., hostname=jenkins-hbase4.apache.org,42467,1681034019544, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-09 09:54:22,485 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:22,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59804 deadline: 1681034122482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:22,493 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:22,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59752 deadline: 1681034122492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:22,566 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:22,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59788 deadline: 1681034122564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:22,642 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=405 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/B/631cae5f1f8e4466898437c28d0a008a 2023-04-09 09:54:22,657 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=405 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/C/6aaaad0d32f3422eabee684a3563ccbb 2023-04-09 09:54:22,668 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/A/deec1cf53c2c478d849a356122f3ee70 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/deec1cf53c2c478d849a356122f3ee70 2023-04-09 09:54:22,675 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/deec1cf53c2c478d849a356122f3ee70, entries=150, sequenceid=405, filesize=25.4 K 2023-04-09 09:54:22,677 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/B/631cae5f1f8e4466898437c28d0a008a as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/631cae5f1f8e4466898437c28d0a008a 2023-04-09 09:54:22,683 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/631cae5f1f8e4466898437c28d0a008a, entries=150, sequenceid=405, filesize=11.9 K 2023-04-09 09:54:22,684 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/C/6aaaad0d32f3422eabee684a3563ccbb as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/6aaaad0d32f3422eabee684a3563ccbb 2023-04-09 09:54:22,689 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/6aaaad0d32f3422eabee684a3563ccbb, entries=150, sequenceid=405, filesize=11.9 K 2023-04-09 09:54:22,690 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for b86d94f6f111f1359f84891af8d12a40 in 503ms, sequenceid=405, compaction requested=false 2023-04-09 09:54:22,690 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b86d94f6f111f1359f84891af8d12a40: 2023-04-09 09:54:22,690 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:22,690 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:22,690 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:22,690 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:22,690 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:22,690 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:22,692 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:22,692 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:22,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:22,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:22,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:22,692 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:22,692 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:22,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:22,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:22,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:22,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:22,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:22,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:22,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9a5c1e2[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:22,695 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:22,695 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:22,696 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:22,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:22,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:22,696 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:22,696 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:22,696 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:22,696 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:22,696 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:22,698 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:22,698 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:22,698 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:22,698 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:22,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:22,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:22,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:22,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:22,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:22,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:22,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:22,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:22,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:22,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:22,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:22,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:22,701 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:22,701 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:22,701 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:22,701 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:22,701 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:22,702 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:22,702 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:22,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:22,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:22,702 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:22,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:22,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:22,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:22,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:22,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:22,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:22,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:22,704 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:22,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:22,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:22,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:22,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:22,705 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:22,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:22,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:22,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:22,707 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:22,707 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:22,707 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:22,707 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:22,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:22,707 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:22,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:22,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:22,707 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:22,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:22,707 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:22,707 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:22,707 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:22,707 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:22,707 DEBUG [Thread-888] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:22,707 DEBUG [Thread-888] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:22,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:22,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:22,807 DEBUG [Thread-888] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:22,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:22,810 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:22,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:22,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:22,813 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:22,813 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:22,813 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:22,813 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:22,813 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:22,813 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:22,815 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:22,815 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:22,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:22,815 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:22,815 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:22,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:22,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:22,816 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:22,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:22,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:22,817 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:22,817 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:22,817 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:22,817 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:22,817 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:22,817 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:22,817 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:22,817 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:22,817 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:22,817 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. started... 2023-04-09 09:54:22,817 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b86d94f6f111f1359f84891af8d12a40 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-09 09:54:22,818 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=A 2023-04-09 09:54:22,818 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:22,818 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=B 2023-04-09 09:54:22,818 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:22,818 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=C 2023-04-09 09:54:22,818 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:22,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on b86d94f6f111f1359f84891af8d12a40 2023-04-09 09:54:22,869 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. as already flushing 2023-04-09 09:54:22,903 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:22,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59788 deadline: 1681034122899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:23,006 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:23,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59788 deadline: 1681034123004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:23,211 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:23,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59788 deadline: 1681034123209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:23,240 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040922836ee8020048719ea2cb212d16450e to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040922836ee8020048719ea2cb212d16450e 2023-04-09 09:54:23,342 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/A/d4212a57fb5b472aaa6bea09d200b4b4 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/d4212a57fb5b472aaa6bea09d200b4b4 2023-04-09 09:54:23,347 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b86d94f6f111f1359f84891af8d12a40/A of b86d94f6f111f1359f84891af8d12a40 into d4212a57fb5b472aaa6bea09d200b4b4(size=26.0 K), total size for store is 51.4 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-09 09:54:23,347 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b86d94f6f111f1359f84891af8d12a40: 2023-04-09 09:54:23,347 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40., storeName=b86d94f6f111f1359f84891af8d12a40/A, priority=13, startTime=1681034062085; duration=1sec 2023-04-09 09:54:23,347 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:54:23,514 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:23,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59788 deadline: 1681034123512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:23,662 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=425, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/A/349bad1c4d7c49fda3a539991da3786d 2023-04-09 09:54:24,017 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:24,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59788 deadline: 1681034124017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:24,076 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=425 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/B/d81935ed9ffd46c7a439588a332b12cf 2023-04-09 09:54:24,487 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=425 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/C/3129816e9c6446468fd588ef4a22f201 2023-04-09 09:54:24,488 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:24,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59804 deadline: 1681034124487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:24,489 DEBUG [Thread-882] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4126 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40., hostname=jenkins-hbase4.apache.org,42467,1681034019544, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-09 09:54:24,494 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/A/349bad1c4d7c49fda3a539991da3786d as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/349bad1c4d7c49fda3a539991da3786d 2023-04-09 09:54:24,498 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/349bad1c4d7c49fda3a539991da3786d, entries=150, sequenceid=425, filesize=25.4 K 2023-04-09 09:54:24,499 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/B/d81935ed9ffd46c7a439588a332b12cf as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/d81935ed9ffd46c7a439588a332b12cf 2023-04-09 09:54:24,504 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/d81935ed9ffd46c7a439588a332b12cf, entries=150, sequenceid=425, filesize=11.9 K 2023-04-09 09:54:24,505 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/C/3129816e9c6446468fd588ef4a22f201 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/3129816e9c6446468fd588ef4a22f201 2023-04-09 09:54:24,511 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/3129816e9c6446468fd588ef4a22f201, entries=150, sequenceid=425, filesize=11.9 K 2023-04-09 09:54:24,512 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for b86d94f6f111f1359f84891af8d12a40 in 1695ms, sequenceid=425, compaction requested=true 2023-04-09 09:54:24,512 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b86d94f6f111f1359f84891af8d12a40: 2023-04-09 09:54:24,512 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:24,512 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:24,512 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:24,512 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:24,512 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:24,512 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:24,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on b86d94f6f111f1359f84891af8d12a40 2023-04-09 09:54:24,516 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:24,516 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:24,516 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b86d94f6f111f1359f84891af8d12a40 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-09 09:54:24,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:24,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:24,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:24,516 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=A 2023-04-09 09:54:24,516 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:24,517 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:24,517 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:24,517 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=B 2023-04-09 09:54:24,517 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:24,517 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=C 2023-04-09 09:54:24,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:24,517 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:24,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:24,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:24,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:24,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:24,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:24,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e4d4882[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:24,519 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:24,519 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:24,520 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:24,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:24,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:24,521 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:24,521 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:24,521 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:24,521 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:24,521 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:24,522 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:24,522 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:24,522 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:24,522 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:24,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:24,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:24,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:24,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:24,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:24,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:24,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:24,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:24,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:24,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:24,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:24,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:24,525 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:24,525 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:24,526 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:24,526 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:24,526 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:24,530 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:24,530 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:24,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:24,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:24,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:24,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:24,531 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:24,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:24,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:24,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:24,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:24,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:24,532 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:24,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:24,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:24,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:24,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:24,534 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:24,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:24,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:24,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:24,537 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:24,537 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:24,537 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:24,537 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:24,537 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:24,537 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:24,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:24,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:24,537 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:24,537 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:24,537 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:24,537 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:24,537 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:24,537 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:24,537 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:24,537 DEBUG [Thread-888] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:24,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:24,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:24,538 DEBUG [Thread-888] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:24,589 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:24,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59752 deadline: 1681034124588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:24,638 DEBUG [Thread-888] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:24,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:24,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:24,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:24,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:24,642 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:24,642 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:24,642 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:24,642 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:24,642 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:24,642 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:24,644 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:24,644 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:24,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:24,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:24,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:24,644 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:24,644 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:24,645 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:24,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:24,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:24,645 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:24,646 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:24,646 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:24,646 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:24,646 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:24,646 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:24,646 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:24,646 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:24,646 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:24,646 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. started... 2023-04-09 09:54:24,646 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. as already flushing 2023-04-09 09:54:24,647 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:24,647 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:24,647 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:24,647 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:24,647 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:24,647 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:24,649 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:24,649 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:24,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:24,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:24,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:24,650 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:24,650 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:24,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:24,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:24,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:24,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:24,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:24,651 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:24,651 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36f1f970[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:24,651 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:24,651 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:24,655 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:24,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:24,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:24,655 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:24,655 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:24,655 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:24,655 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:24,656 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:24,657 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:24,657 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:24,657 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:24,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:24,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:24,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:24,657 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:24,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:24,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:24,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:24,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:24,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:24,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:24,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:24,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:24,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:24,659 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:24,659 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:24,659 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:24,659 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:24,659 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:24,662 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:24,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:24,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:24,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:24,662 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:24,662 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:24,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:24,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:24,662 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:24,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:24,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:24,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:24,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:24,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:24,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:24,664 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:24,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:24,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:24,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:24,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:24,666 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:24,666 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:24,667 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:24,667 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:24,667 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:24,667 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:24,667 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:24,667 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:24,667 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:24,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:24,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:24,667 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:24,667 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:24,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:24,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:24,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:24,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:24,668 DEBUG [Thread-888] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:24,668 DEBUG [Thread-888] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:24,691 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:24,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59752 deadline: 1681034124690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:24,768 DEBUG [Thread-888] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:24,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:24,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:24,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:24,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:24,772 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:24,772 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:24,772 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:24,772 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:24,772 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:24,772 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:24,778 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:24,778 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:24,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:24,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:24,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:24,779 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:24,779 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:24,779 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:24,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:24,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:24,780 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:24,780 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:24,780 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:24,780 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:24,780 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:24,780 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:24,780 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:24,780 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:24,780 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:24,781 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. started... 2023-04-09 09:54:24,781 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. as already flushing 2023-04-09 09:54:24,781 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:24,781 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:24,781 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:24,781 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:24,781 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:24,781 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:24,782 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:24,782 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:24,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:24,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:24,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:24,783 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:24,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:24,783 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:24,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:24,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:24,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:24,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:24,784 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:24,784 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:24,784 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7eccaf48[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:24,784 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:24,785 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:24,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:24,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:24,786 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:24,786 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:24,786 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:24,786 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:24,786 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:24,787 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:24,787 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:24,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:24,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:24,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:24,787 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:24,787 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:24,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:24,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:24,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:24,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:24,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:24,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:24,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:24,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:24,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:24,790 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:24,790 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:24,790 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:24,790 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:24,790 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:24,792 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:24,792 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:24,792 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:24,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:24,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:24,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:24,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:24,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:24,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:24,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:24,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:24,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:24,799 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:24,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:24,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:24,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:24,801 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:24,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:24,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:24,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:24,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:24,803 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:24,803 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:24,803 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:24,803 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:24,803 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:24,803 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:24,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:24,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:24,803 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:24,803 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:24,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:24,804 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:24,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:24,804 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:24,804 DEBUG [Thread-888] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:24,804 DEBUG [Thread-888] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:24,804 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:24,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:24,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:24,893 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:24,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59752 deadline: 1681034124893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:24,904 DEBUG [Thread-888] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:24,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:24,907 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:24,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:24,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:24,917 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:24,917 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:24,917 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:24,917 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:24,918 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:24,918 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:24,920 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:24,920 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:24,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:24,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:24,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:24,921 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:24,921 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:24,921 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:24,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:24,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:24,922 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:24,932 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:24,932 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:24,932 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:24,932 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:24,932 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:24,932 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:24,932 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:24,933 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:24,933 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. started... 2023-04-09 09:54:24,933 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. as already flushing 2023-04-09 09:54:24,933 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:24,934 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:24,934 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:24,934 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:24,934 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:24,934 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:24,987 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:24,988 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:24,988 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:24,992 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:24,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:24,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:24,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:24,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:24,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:24,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:24,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:24,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:24,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:24,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5941dba2[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:24,995 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:24,995 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:24,997 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304095090d3e948a44824bf89b17a6745dab4 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304095090d3e948a44824bf89b17a6745dab4 2023-04-09 09:54:25,000 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:25,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:25,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:25,000 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:25,000 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:25,000 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:25,003 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,003 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:25,003 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:25,003 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:25,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:25,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:25,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:25,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:25,004 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,004 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:25,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:25,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:25,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:25,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:25,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:25,006 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:25,006 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:25,006 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:25,006 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:25,006 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:25,008 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:25,008 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:25,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:25,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:25,008 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:25,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:25,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:25,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:25,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:25,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:25,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:25,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:25,009 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:25,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:25,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:25,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,011 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:25,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:25,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,015 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:25,016 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:25,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:25,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:25,016 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,016 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:25,016 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:25,016 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:25,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:25,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:25,016 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,016 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:25,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:25,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:25,016 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:25,016 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:25,016 DEBUG [Thread-888] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:25,016 DEBUG [Thread-888] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:25,024 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:25,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59788 deadline: 1681034125022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,043 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=439, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/A/3197f69ce7934090a7dfb01887e038dd 2023-04-09 09:54:25,083 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=439 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/B/dfbb757ad37546abbadd5f72b1265342 2023-04-09 09:54:25,117 DEBUG [Thread-888] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:25,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:25,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:25,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:25,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:25,125 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:25,125 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:25,126 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:25,126 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:25,126 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:25,126 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:25,129 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,129 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:25,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:25,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:25,129 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,129 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:25,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:25,130 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:25,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:25,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:25,131 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:25,131 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:25,131 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:25,132 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:25,132 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:25,132 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:25,140 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:25,140 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:25,140 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:25,140 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. started... 2023-04-09 09:54:25,141 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. as already flushing 2023-04-09 09:54:25,141 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:25,141 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:25,141 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:25,141 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,141 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:25,141 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:25,142 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:25,143 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:25,143 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:25,144 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:25,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:25,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:25,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:25,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:25,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:25,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:25,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5dcc8150[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:25,149 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:25,149 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:25,151 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:25,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:25,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:25,151 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:25,151 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:25,151 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:25,151 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,151 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:25,153 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:25,153 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:25,153 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:25,153 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:25,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:25,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:25,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:25,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:25,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:25,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:25,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:25,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:25,156 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:25,156 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:25,156 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:25,156 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:25,156 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:25,160 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:25,160 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:25,160 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:25,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:25,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:25,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:25,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:25,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:25,161 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:25,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:25,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:25,162 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:25,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:25,165 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:25,165 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:25,165 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:25,165 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:25,165 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,165 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:25,165 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:25,165 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:25,165 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:25,165 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:25,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:25,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:25,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:25,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:25,166 DEBUG [Thread-888] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:25,166 DEBUG [Thread-888] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:25,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:25,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:25,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:25,168 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:25,195 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:25,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59752 deadline: 1681034125194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,266 DEBUG [Thread-888] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:25,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:25,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:25,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:25,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:25,269 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:25,270 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:25,270 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:25,270 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:25,270 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:25,270 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:25,272 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,272 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:25,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:25,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:25,272 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,272 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:25,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:25,272 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:25,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:25,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:25,273 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:25,273 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:25,273 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:25,273 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:25,273 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:25,273 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:25,273 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:25,273 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:25,273 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:25,274 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. started... 2023-04-09 09:54:25,274 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. as already flushing 2023-04-09 09:54:25,274 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:25,274 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:25,274 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:25,274 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,274 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:25,274 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:25,275 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:25,275 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:25,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:25,276 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:25,276 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:25,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:25,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:25,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:25,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:25,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:25,277 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:25,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64cfd9e0[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:25,277 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:25,280 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:25,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:25,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:25,280 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:25,280 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:25,280 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,280 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:25,280 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:25,282 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,282 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:25,282 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:25,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:25,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:25,282 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:25,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:25,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:25,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:25,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:25,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:25,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:25,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:25,284 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:25,284 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:25,284 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:25,284 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:25,284 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:25,286 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:25,286 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:25,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:25,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:25,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:25,286 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:25,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:25,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:25,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:25,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:25,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:25,287 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:25,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:25,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:25,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:25,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,289 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:25,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:25,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,291 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:25,292 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:25,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:25,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:25,292 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,292 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:25,292 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:25,292 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,292 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:25,292 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:25,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:25,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:25,292 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:25,292 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:25,292 DEBUG [Thread-888] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:25,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:25,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:25,292 DEBUG [Thread-888] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:25,392 DEBUG [Thread-888] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:25,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:25,395 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:25,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:25,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:25,396 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:25,396 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:25,396 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:25,396 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:25,397 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:25,397 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:25,399 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,399 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:25,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:25,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:25,399 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,399 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:25,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:25,400 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:25,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:25,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:25,400 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:25,400 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:25,400 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:25,400 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:25,401 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:25,401 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:25,401 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:25,401 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:25,401 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:25,401 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. started... 2023-04-09 09:54:25,401 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. as already flushing 2023-04-09 09:54:25,401 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:25,401 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:25,401 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:25,401 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,401 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:25,401 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:25,403 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:25,403 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:25,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:25,403 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:25,403 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:25,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:25,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:25,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:25,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:25,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:25,405 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:25,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f629cde[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:25,405 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:25,406 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:25,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:25,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:25,406 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:25,406 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:25,406 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:25,406 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,406 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:25,408 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:25,408 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:25,408 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:25,408 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:25,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:25,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:25,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:25,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:25,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:25,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:25,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:25,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:25,411 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:25,411 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:25,411 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:25,411 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:25,411 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:25,416 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:25,416 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:25,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:25,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:25,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:25,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:25,416 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:25,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:25,416 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:25,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:25,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:25,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:25,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:25,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:25,417 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:25,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:25,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:25,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,422 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,422 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:25,422 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:25,422 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:25,422 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,422 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:25,422 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:25,422 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:25,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:25,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:25,423 DEBUG [Thread-888] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:25,423 DEBUG [Thread-888] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:25,423 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:25,423 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:25,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:25,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:25,424 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:25,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:25,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:25,511 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=439 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/C/0bea334c8be04d178e574792ddf3a786 2023-04-09 09:54:25,517 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/A/3197f69ce7934090a7dfb01887e038dd as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/3197f69ce7934090a7dfb01887e038dd 2023-04-09 09:54:25,522 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/3197f69ce7934090a7dfb01887e038dd, entries=150, sequenceid=439, filesize=25.4 K 2023-04-09 09:54:25,523 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/B/dfbb757ad37546abbadd5f72b1265342 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/dfbb757ad37546abbadd5f72b1265342 2023-04-09 09:54:25,523 DEBUG [Thread-888] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:25,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:25,525 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:25,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:25,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:25,526 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:25,526 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:25,526 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:25,526 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:25,527 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:25,527 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:25,528 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,528 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:25,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:25,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:25,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:25,529 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,529 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:25,529 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:25,529 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/dfbb757ad37546abbadd5f72b1265342, entries=150, sequenceid=439, filesize=11.9 K 2023-04-09 09:54:25,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:25,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:25,530 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:25,530 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:25,530 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:25,530 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:25,530 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:25,530 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:25,531 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:25,531 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:25,531 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:25,531 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. started... 2023-04-09 09:54:25,531 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/C/0bea334c8be04d178e574792ddf3a786 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/0bea334c8be04d178e574792ddf3a786 2023-04-09 09:54:25,531 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. as already flushing 2023-04-09 09:54:25,531 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:25,531 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:25,531 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:25,531 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,531 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:25,531 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:25,540 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:25,540 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:25,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:25,540 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:25,540 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:25,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:25,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:25,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:25,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:25,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:25,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28a9d45b[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:25,543 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:25,543 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:25,546 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:25,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:25,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:25,546 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:25,546 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:25,546 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:25,547 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/0bea334c8be04d178e574792ddf3a786, entries=150, sequenceid=439, filesize=11.9 K 2023-04-09 09:54:25,548 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:25,548 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:25,549 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:25,549 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,549 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for b86d94f6f111f1359f84891af8d12a40 in 1033ms, sequenceid=439, compaction requested=true 2023-04-09 09:54:25,549 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b86d94f6f111f1359f84891af8d12a40: 2023-04-09 09:54:25,549 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:54:25,549 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-09 09:54:25,549 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-09 09:54:25,549 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:54:25,549 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:25,549 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:25,549 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:54:25,549 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:25,549 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:25,549 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:25,549 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:25,550 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:25,551 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 104730 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-09 09:54:25,551 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49290 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-09 09:54:25,551 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1912): b86d94f6f111f1359f84891af8d12a40/A is initiating minor compaction (all files) 2023-04-09 09:54:25,551 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): b86d94f6f111f1359f84891af8d12a40/B is initiating minor compaction (all files) 2023-04-09 09:54:25,551 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b86d94f6f111f1359f84891af8d12a40/A in TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:25,551 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b86d94f6f111f1359f84891af8d12a40/B in TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:25,551 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/d4212a57fb5b472aaa6bea09d200b4b4, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/deec1cf53c2c478d849a356122f3ee70, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/349bad1c4d7c49fda3a539991da3786d, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/3197f69ce7934090a7dfb01887e038dd] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp, totalSize=102.3 K 2023-04-09 09:54:25,551 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/0dcd174fd5394f7eb7fe53a42910ce19, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/631cae5f1f8e4466898437c28d0a008a, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/d81935ed9ffd46c7a439588a332b12cf, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/dfbb757ad37546abbadd5f72b1265342] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp, totalSize=48.1 K 2023-04-09 09:54:25,552 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting d4212a57fb5b472aaa6bea09d200b4b4, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=391, earliestPutTs=1681034060356 2023-04-09 09:54:25,552 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 0dcd174fd5394f7eb7fe53a42910ce19, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=391, earliestPutTs=1681034060356 2023-04-09 09:54:25,552 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting deec1cf53c2c478d849a356122f3ee70, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=405, earliestPutTs=1681034061590 2023-04-09 09:54:25,553 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 631cae5f1f8e4466898437c28d0a008a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=405, earliestPutTs=1681034061590 2023-04-09 09:54:25,553 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 349bad1c4d7c49fda3a539991da3786d, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=425, earliestPutTs=1681034062254 2023-04-09 09:54:25,553 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting d81935ed9ffd46c7a439588a332b12cf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=425, earliestPutTs=1681034062254 2023-04-09 09:54:25,554 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 3197f69ce7934090a7dfb01887e038dd, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=439, earliestPutTs=1681034062882 2023-04-09 09:54:25,554 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting dfbb757ad37546abbadd5f72b1265342, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=439, earliestPutTs=1681034062882 2023-04-09 09:54:25,557 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:25,558 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:25,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:25,558 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:25,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:25,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:25,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:25,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:25,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:25,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:25,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:25,559 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:25,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:25,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:25,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:25,562 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:25,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:25,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,573 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,573 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:25,573 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:25,573 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:25,573 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:25,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:25,573 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:25,573 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:25,573 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:25,573 DEBUG [Thread-888] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:25,573 DEBUG [Thread-888] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:25,573 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:25,574 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): b86d94f6f111f1359f84891af8d12a40#B#compaction#175 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:54:25,575 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:25,575 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:25,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:25,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:25,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:25,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:25,586 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] throttle.PressureAwareThroughputController(145): b86d94f6f111f1359f84891af8d12a40#A#compaction#174 average throughput is 1.41 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:54:25,653 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/B/95e411e7f0544694b3de909e80e7f789 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/95e411e7f0544694b3de909e80e7f789 2023-04-09 09:54:25,659 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b86d94f6f111f1359f84891af8d12a40/B of b86d94f6f111f1359f84891af8d12a40 into 95e411e7f0544694b3de909e80e7f789(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:54:25,660 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b86d94f6f111f1359f84891af8d12a40: 2023-04-09 09:54:25,660 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40., storeName=b86d94f6f111f1359f84891af8d12a40/B, priority=12, startTime=1681034065549; duration=0sec 2023-04-09 09:54:25,660 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:54:25,660 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-09 09:54:25,661 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49290 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-09 09:54:25,662 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): b86d94f6f111f1359f84891af8d12a40/C is initiating minor compaction (all files) 2023-04-09 09:54:25,662 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b86d94f6f111f1359f84891af8d12a40/C in TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:25,662 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/e5f9f351093d47979a26310cec35f896, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/6aaaad0d32f3422eabee684a3563ccbb, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/3129816e9c6446468fd588ef4a22f201, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/0bea334c8be04d178e574792ddf3a786] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp, totalSize=48.1 K 2023-04-09 09:54:25,662 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting e5f9f351093d47979a26310cec35f896, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=391, earliestPutTs=1681034060356 2023-04-09 09:54:25,663 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 6aaaad0d32f3422eabee684a3563ccbb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=405, earliestPutTs=1681034061590 2023-04-09 09:54:25,663 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 3129816e9c6446468fd588ef4a22f201, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=425, earliestPutTs=1681034062254 2023-04-09 09:54:25,664 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 0bea334c8be04d178e574792ddf3a786, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=439, earliestPutTs=1681034062882 2023-04-09 09:54:25,673 DEBUG [Thread-888] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:25,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:25,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:25,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:25,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:25,679 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:25,679 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:25,679 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:25,679 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:25,680 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): b86d94f6f111f1359f84891af8d12a40#C#compaction#176 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:54:25,680 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:25,680 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:25,682 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,683 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:25,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:25,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:25,683 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,683 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:25,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:25,684 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:25,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:25,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:25,684 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:25,685 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:25,685 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:25,685 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:25,685 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:25,685 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:25,685 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:25,685 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:25,686 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:25,686 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. started... 2023-04-09 09:54:25,686 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b86d94f6f111f1359f84891af8d12a40 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-09 09:54:25,686 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=A 2023-04-09 09:54:25,686 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:25,687 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=B 2023-04-09 09:54:25,687 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:25,687 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=C 2023-04-09 09:54:25,687 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:25,697 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/A/3e62ad31d3a84cce99361b410a12aaa4 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/3e62ad31d3a84cce99361b410a12aaa4 2023-04-09 09:54:25,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on b86d94f6f111f1359f84891af8d12a40 2023-04-09 09:54:25,705 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. as already flushing 2023-04-09 09:54:25,705 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/C/4eace68c8a0d43b984cf9d16e4257de6 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/4eace68c8a0d43b984cf9d16e4257de6 2023-04-09 09:54:25,711 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b86d94f6f111f1359f84891af8d12a40/A of b86d94f6f111f1359f84891af8d12a40 into 3e62ad31d3a84cce99361b410a12aaa4(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:54:25,711 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b86d94f6f111f1359f84891af8d12a40: 2023-04-09 09:54:25,711 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40., storeName=b86d94f6f111f1359f84891af8d12a40/A, priority=12, startTime=1681034065549; duration=0sec 2023-04-09 09:54:25,711 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:54:25,716 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b86d94f6f111f1359f84891af8d12a40/C of b86d94f6f111f1359f84891af8d12a40 into 4eace68c8a0d43b984cf9d16e4257de6(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:54:25,716 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b86d94f6f111f1359f84891af8d12a40: 2023-04-09 09:54:25,716 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40., storeName=b86d94f6f111f1359f84891af8d12a40/C, priority=12, startTime=1681034065549; duration=0sec 2023-04-09 09:54:25,717 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:54:25,724 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230409f6afd67fd9c84dca97d48373fc0a6a4a to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409f6afd67fd9c84dca97d48373fc0a6a4a 2023-04-09 09:54:25,739 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=459, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/A/b5e1d07360704206927465f48f02c98b 2023-04-09 09:54:25,741 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:25,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59752 deadline: 1681034125740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:25,846 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:25,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59752 deadline: 1681034125843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:26,049 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:26,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59752 deadline: 1681034126048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:26,155 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=459 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/B/17f4fc67a0e24903832774556a38b515 2023-04-09 09:54:26,167 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=459 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/C/36605c84a13449948e3a9ac065bc563e 2023-04-09 09:54:26,175 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/A/b5e1d07360704206927465f48f02c98b as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/b5e1d07360704206927465f48f02c98b 2023-04-09 09:54:26,179 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/b5e1d07360704206927465f48f02c98b, entries=150, sequenceid=459, filesize=25.4 K 2023-04-09 09:54:26,180 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/B/17f4fc67a0e24903832774556a38b515 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/17f4fc67a0e24903832774556a38b515 2023-04-09 09:54:26,184 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/17f4fc67a0e24903832774556a38b515, entries=150, sequenceid=459, filesize=11.9 K 2023-04-09 09:54:26,185 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/C/36605c84a13449948e3a9ac065bc563e as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/36605c84a13449948e3a9ac065bc563e 2023-04-09 09:54:26,189 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/36605c84a13449948e3a9ac065bc563e, entries=150, sequenceid=459, filesize=11.9 K 2023-04-09 09:54:26,190 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for b86d94f6f111f1359f84891af8d12a40 in 504ms, sequenceid=459, compaction requested=false 2023-04-09 09:54:26,190 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b86d94f6f111f1359f84891af8d12a40: 2023-04-09 09:54:26,190 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:26,190 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:26,190 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:26,190 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:26,190 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:26,191 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:26,193 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:26,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:26,193 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:26,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:26,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:26,194 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:26,194 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:26,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:26,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:26,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:26,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:26,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:26,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:26,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11d6061f[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:26,195 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:26,196 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:26,197 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:26,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:26,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:26,197 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:26,197 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:26,197 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:26,197 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:26,197 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:26,198 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:26,198 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:26,198 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:26,198 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:26,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:26,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:26,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:26,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:26,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:26,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:26,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:26,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:26,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:26,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:26,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:26,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:26,201 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:26,201 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:26,201 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:26,201 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:26,201 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:26,203 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:26,203 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:26,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:26,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:26,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:26,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:26,204 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:26,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:26,205 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:26,207 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:26,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:26,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:26,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:26,207 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:26,207 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:26,207 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:26,207 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:26,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:26,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:26,207 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:26,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:26,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:26,207 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:26,207 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:26,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:26,208 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:26,208 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:26,208 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:26,208 DEBUG [Thread-888] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:26,208 DEBUG [Thread-888] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:26,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:26,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:26,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:26,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:26,209 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:26,308 DEBUG [Thread-888] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:26,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:26,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:26,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:26,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:26,311 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:26,311 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:26,311 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:26,311 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:26,312 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:26,312 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:26,313 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:26,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:26,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:26,313 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:26,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:26,314 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:26,314 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:26,314 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:26,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:26,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:26,315 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:26,315 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:26,315 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:26,315 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:26,315 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:26,315 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:26,315 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:26,315 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:26,315 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:26,316 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. started... 2023-04-09 09:54:26,316 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b86d94f6f111f1359f84891af8d12a40 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-09 09:54:26,316 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=A 2023-04-09 09:54:26,316 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:26,316 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=B 2023-04-09 09:54:26,316 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:26,316 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=C 2023-04-09 09:54:26,316 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:26,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on b86d94f6f111f1359f84891af8d12a40 2023-04-09 09:54:26,352 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. as already flushing 2023-04-09 09:54:26,383 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:26,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59752 deadline: 1681034126383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:26,486 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:26,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59752 deadline: 1681034126484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:26,503 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:26,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59766 deadline: 1681034126502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:26,505 DEBUG [Thread-880] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8166 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40., hostname=jenkins-hbase4.apache.org,42467,1681034019544, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-09 09:54:26,506 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:26,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59822 deadline: 1681034126505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:26,508 DEBUG [Thread-878] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8169 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40., hostname=jenkins-hbase4.apache.org,42467,1681034019544, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-09 09:54:26,689 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:26,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59752 deadline: 1681034126687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:26,729 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304097f9a45b3867f49d39f3f3d7d8599b902 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304097f9a45b3867f49d39f3f3d7d8599b902 2023-04-09 09:54:26,991 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:26,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59752 deadline: 1681034126991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:27,043 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:27,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59788 deadline: 1681034127042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:27,045 DEBUG [Thread-884] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4145 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40., hostname=jenkins-hbase4.apache.org,42467,1681034019544, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-09 09:54:27,135 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=474, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/A/29284701768c44d7bec2c846c4f665cc 2023-04-09 09:54:27,149 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=474 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/B/434756787fbb49c2919d209e1d3d0bba 2023-04-09 09:54:27,495 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:27,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59752 deadline: 1681034127493, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:27,560 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=474 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/C/d30c2449774d483392e3b42c577abc35 2023-04-09 09:54:27,565 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/A/29284701768c44d7bec2c846c4f665cc as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/29284701768c44d7bec2c846c4f665cc 2023-04-09 09:54:27,569 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/29284701768c44d7bec2c846c4f665cc, entries=150, sequenceid=474, filesize=25.4 K 2023-04-09 09:54:27,570 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/B/434756787fbb49c2919d209e1d3d0bba as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/434756787fbb49c2919d209e1d3d0bba 2023-04-09 09:54:27,574 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/434756787fbb49c2919d209e1d3d0bba, entries=150, sequenceid=474, filesize=11.9 K 2023-04-09 09:54:27,575 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/C/d30c2449774d483392e3b42c577abc35 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/d30c2449774d483392e3b42c577abc35 2023-04-09 09:54:27,581 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/d30c2449774d483392e3b42c577abc35, entries=150, sequenceid=474, filesize=11.9 K 2023-04-09 09:54:27,582 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for b86d94f6f111f1359f84891af8d12a40 in 1265ms, sequenceid=474, compaction requested=true 2023-04-09 09:54:27,582 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b86d94f6f111f1359f84891af8d12a40: 2023-04-09 09:54:27,582 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:27,582 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:27,582 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:27,582 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:27,582 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:27,582 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:27,584 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:27,584 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:27,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:27,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:27,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:27,584 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:27,584 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:27,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:27,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:27,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:27,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:27,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:27,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:27,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b4cdd1c[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:27,586 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:27,586 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:27,588 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:27,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:27,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:27,588 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:27,588 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:27,588 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:27,588 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:27,588 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:27,589 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:27,589 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:27,589 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:27,589 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:27,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:27,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:27,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:27,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:27,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:27,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:27,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:27,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:27,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:27,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:27,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:27,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:27,592 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:27,592 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:27,592 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:27,592 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:27,592 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:27,594 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:27,594 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:27,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:27,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:27,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:27,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:27,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:27,594 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:27,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:27,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:27,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:27,595 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:27,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:27,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:27,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:27,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:27,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:27,596 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:27,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:27,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:27,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:27,601 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:27,601 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:27,601 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:27,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:27,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:27,601 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:27,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:27,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:27,601 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:27,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:27,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:27,601 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:27,601 DEBUG [Thread-888] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:27,601 DEBUG [Thread-888] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:27,601 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:27,602 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:27,602 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:27,602 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:27,702 DEBUG [Thread-888] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:27,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:27,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:27,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:27,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:27,705 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:27,705 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:27,705 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:27,705 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:27,705 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:27,706 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:27,707 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:27,707 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:27,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:27,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:27,707 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:27,708 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:27,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:27,708 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:27,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:27,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:27,709 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:27,709 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:27,709 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:27,709 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:27,709 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:27,709 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:27,709 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:27,709 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:27,709 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:27,709 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. started... 2023-04-09 09:54:27,709 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b86d94f6f111f1359f84891af8d12a40 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-09 09:54:27,709 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=A 2023-04-09 09:54:27,710 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:27,710 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=B 2023-04-09 09:54:27,710 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:27,710 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=C 2023-04-09 09:54:27,710 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:27,724 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304098750ea7ff0ad4200b9985ae8cb02746f to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304098750ea7ff0ad4200b9985ae8cb02746f 2023-04-09 09:54:28,138 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=493, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/A/84c6441be2e446fdaf879320e18877f7 2023-04-09 09:54:28,150 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=493 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/B/d6d30be793df4c6080a0736951d3bb0c 2023-04-09 09:54:28,177 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=493 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/C/e3b74271b37f4154ae0bacf5c7924761 2023-04-09 09:54:28,181 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/A/84c6441be2e446fdaf879320e18877f7 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/84c6441be2e446fdaf879320e18877f7 2023-04-09 09:54:28,186 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/84c6441be2e446fdaf879320e18877f7, entries=150, sequenceid=493, filesize=25.4 K 2023-04-09 09:54:28,187 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/B/d6d30be793df4c6080a0736951d3bb0c as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/d6d30be793df4c6080a0736951d3bb0c 2023-04-09 09:54:28,192 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/d6d30be793df4c6080a0736951d3bb0c, entries=150, sequenceid=493, filesize=11.9 K 2023-04-09 09:54:28,193 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/C/e3b74271b37f4154ae0bacf5c7924761 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/e3b74271b37f4154ae0bacf5c7924761 2023-04-09 09:54:28,197 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/e3b74271b37f4154ae0bacf5c7924761, entries=150, sequenceid=493, filesize=11.9 K 2023-04-09 09:54:28,198 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=0 B/0 for b86d94f6f111f1359f84891af8d12a40 in 489ms, sequenceid=493, compaction requested=true 2023-04-09 09:54:28,198 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b86d94f6f111f1359f84891af8d12a40: 2023-04-09 09:54:28,198 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:28,198 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:28,198 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:28,198 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,200 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:28,200 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:28,201 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,201 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:28,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:28,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:28,202 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:28,202 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:28,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:28,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:28,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:28,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:28,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:28,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@602324c7[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:28,204 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:28,204 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:28,205 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:28,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:28,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:28,205 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:28,205 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:28,205 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:28,206 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,206 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:28,211 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:28,211 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:28,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:28,211 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:28,212 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:28,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:28,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:28,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:28,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:28,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:28,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:28,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:28,214 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:28,214 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:28,214 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:28,214 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:28,214 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:28,216 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:28,216 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:28,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:28,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:28,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:28,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:28,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:28,216 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:28,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:28,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:28,217 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:28,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:28,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:28,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:28,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:28,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,218 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:28,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:28,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,228 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,228 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:28,229 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:28,228 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:28,229 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:28,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:28,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:28,229 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,229 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:28,229 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:28,229 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:28,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:28,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:28,229 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:28,229 DEBUG [Thread-888] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:28,229 DEBUG [Thread-888] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:28,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:28,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:28,329 DEBUG [Thread-888] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:28,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:28,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:28,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:28,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:28,335 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:28,335 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:28,335 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:28,335 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:28,335 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:28,335 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:28,337 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:28,337 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:28,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:28,337 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,337 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:28,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:28,338 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:28,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:28,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:28,338 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:28,338 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:28,338 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:28,339 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:28,339 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:28,339 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:28,339 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:28,339 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:28,339 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:28,339 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. started... 2023-04-09 09:54:28,340 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b86d94f6f111f1359f84891af8d12a40: 2023-04-09 09:54:28,340 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:28,340 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:28,340 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:28,340 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,340 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:28,340 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:28,343 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,343 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:28,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:28,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:28,344 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:28,344 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:28,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:28,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:28,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:28,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:28,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:28,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ce2f94[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:28,346 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:28,346 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:28,347 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:28,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:28,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:28,348 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:28,348 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:28,348 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:28,348 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,348 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:28,349 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:28,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:28,349 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:28,349 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:28,349 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:28,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:28,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:28,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:28,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:28,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:28,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:28,352 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:28,352 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:28,352 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:28,352 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:28,353 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:28,353 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:28,354 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:28,354 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:28,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:28,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:28,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:28,354 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:28,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:28,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:28,355 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:28,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:28,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:28,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:28,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:28,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:28,356 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:28,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:28,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:28,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,360 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:28,360 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:28,360 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:28,360 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:28,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:28,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:28,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:28,360 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:28,360 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:28,360 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,361 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:28,361 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:28,361 DEBUG [Thread-888] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:28,361 DEBUG [Thread-888] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:28,361 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:28,361 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:28,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:28,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:28,461 DEBUG [Thread-888] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:28,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:28,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:28,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:28,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:28,476 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:28,476 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:28,476 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:28,476 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:28,477 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:28,477 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:28,478 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:28,478 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:28,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:28,478 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:28,478 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:28,479 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:28,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:28,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:28,479 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:28,480 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:28,480 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:28,480 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:28,480 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:28,480 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:28,480 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:28,480 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:28,481 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:28,481 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. started... 2023-04-09 09:54:28,482 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b86d94f6f111f1359f84891af8d12a40: 2023-04-09 09:54:28,482 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:28,482 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:28,482 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:28,482 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,482 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:28,482 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:28,483 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,483 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:28,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:28,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:28,484 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:28,484 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:28,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:28,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:28,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:28,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:28,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:28,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@278927ac[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:28,486 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:28,486 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:28,487 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:28,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:28,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:28,487 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:28,487 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:28,487 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,487 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:28,487 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:28,490 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:28,490 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:28,490 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:28,490 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:28,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:28,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:28,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:28,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:28,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:28,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:28,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:28,493 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:28,493 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:28,493 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:28,493 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:28,493 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:28,493 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:28,495 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:28,495 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:28,495 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:28,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:28,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:28,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:28,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:28,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:28,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:28,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:28,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:28,496 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:28,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:28,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:28,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:28,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,497 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:28,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:28,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,502 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:28,502 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:28,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:28,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:28,502 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:28,502 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,502 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:28,502 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:28,502 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:28,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:28,502 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:28,502 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:28,503 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,503 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:28,503 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:28,503 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:28,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:28,503 DEBUG [Thread-888] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:28,503 DEBUG [Thread-888] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:28,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on b86d94f6f111f1359f84891af8d12a40 2023-04-09 09:54:28,562 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b86d94f6f111f1359f84891af8d12a40 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-09 09:54:28,562 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=A 2023-04-09 09:54:28,563 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:28,563 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=B 2023-04-09 09:54:28,563 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:28,563 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=C 2023-04-09 09:54:28,563 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:28,590 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040918844ff1f3a747e1bae9db30d6b215d7 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040918844ff1f3a747e1bae9db30d6b215d7 2023-04-09 09:54:28,603 DEBUG [Thread-888] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:28,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:28,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:28,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:28,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:28,609 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:28,609 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:28,609 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:28,609 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:28,609 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:28,609 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:28,611 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,611 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:28,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:28,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:28,612 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:28,612 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:28,612 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:28,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:28,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:28,613 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:28,613 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:28,613 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:28,613 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:28,613 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:28,613 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:28,613 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:28,613 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:28,613 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:28,613 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. started... 2023-04-09 09:54:28,614 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. as already flushing 2023-04-09 09:54:28,614 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:28,614 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:28,614 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:28,614 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,614 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:28,614 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:28,615 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,615 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:28,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:28,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:28,615 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:28,615 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:28,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:28,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:28,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:28,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:28,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:28,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ac0ba7c[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:28,618 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:28,618 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:28,619 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:28,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:28,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:28,619 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:28,619 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:28,619 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:28,619 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,619 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:28,620 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:28,620 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:28,621 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:28,621 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:28,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:28,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:28,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:28,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:28,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:28,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:28,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:28,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:28,624 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:28,624 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:28,624 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:28,624 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:28,624 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:28,626 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:28,626 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:28,627 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:28,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:28,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:28,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:28,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:28,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:28,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:28,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:28,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:28,627 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:28,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:28,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:28,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:28,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:28,629 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:28,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,631 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,631 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:28,631 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:28,631 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:28,631 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:28,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:28,632 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:28,632 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:28,632 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:28,632 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:28,632 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:28,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:28,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:28,632 DEBUG [Thread-888] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:28,632 DEBUG [Thread-888] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:28,633 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681034068621, max timestamp=1681034068631Num uniques 859; 2023-04-09 09:54:28,633 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-09 09:54:28,634 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681034068621, max timestamp=1681034068631Num uniques 859; 2023-04-09 09:54:28,634 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-09 09:54:28,634 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681034068621, max timestamp=1681034068631Num uniques 859; 2023-04-09 09:54:28,634 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-09 09:54:28,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:28,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:28,670 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:28,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59804 deadline: 1681034128666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,670 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:28,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59752 deadline: 1681034128667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,732 DEBUG [Thread-888] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:28,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:28,734 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:28,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:28,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:28,736 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:28,736 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:28,736 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:28,736 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:28,736 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:28,736 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:28,738 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:28,738 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:28,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:28,738 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,738 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:28,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:28,738 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:28,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:28,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:28,739 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:28,739 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:28,739 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:28,739 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:28,739 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:28,739 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:28,740 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:28,740 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:28,740 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:28,740 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. started... 2023-04-09 09:54:28,740 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. as already flushing 2023-04-09 09:54:28,740 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:28,740 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:28,740 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:28,740 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,740 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:28,740 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:28,741 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,741 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:28,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:28,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:28,742 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:28,742 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:28,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:28,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:28,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:28,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:28,743 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:28,743 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:28,743 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33bcbaff[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:28,743 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:28,746 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:28,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:28,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:28,746 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:28,746 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:28,746 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:28,746 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,746 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:28,747 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:28,747 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:28,747 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:28,748 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:28,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:28,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:28,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:28,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:28,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:28,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:28,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:28,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:28,751 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:28,751 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:28,751 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:28,751 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:28,751 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:28,753 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:28,753 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:28,754 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:28,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:28,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:28,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:28,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:28,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:28,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:28,754 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:28,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:28,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:28,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:28,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:28,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:28,755 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:28,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:28,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,758 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,758 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:28,758 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:28,758 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,758 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:28,758 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:28,758 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:28,758 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:28,758 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:28,758 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:28,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:28,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:28,758 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:28,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:28,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:28,759 DEBUG [Thread-888] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:28,759 DEBUG [Thread-888] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:28,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:28,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:28,773 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:28,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59804 deadline: 1681034128771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,773 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:28,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59752 deadline: 1681034128771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,859 DEBUG [Thread-888] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:28,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:28,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:28,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:28,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:28,862 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:28,862 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:28,862 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:28,862 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:28,863 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:28,863 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:28,865 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:28,865 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:28,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:28,865 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,865 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:28,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:28,866 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:28,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:28,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:28,866 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:28,866 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:28,867 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:28,867 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:28,867 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:28,867 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:28,868 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:28,868 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:28,868 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:28,868 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. started... 2023-04-09 09:54:28,868 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. as already flushing 2023-04-09 09:54:28,868 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:28,868 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:28,868 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:28,868 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,868 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:28,868 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:28,870 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,870 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:28,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:28,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:28,870 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:28,870 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:28,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:28,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:28,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:28,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:28,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:28,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44f99193[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:28,872 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:28,872 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:28,873 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:28,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:28,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:28,873 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:28,873 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:28,873 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,873 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:28,873 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:28,875 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,875 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:28,875 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:28,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:28,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:28,875 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:28,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:28,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:28,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:28,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:28,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:28,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:28,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:28,877 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:28,877 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:28,878 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:28,878 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:28,878 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:28,880 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:28,880 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:28,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:28,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:28,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:28,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:28,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:28,880 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:28,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:28,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:28,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:28,881 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:28,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:28,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:28,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:28,882 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:28,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:28,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,885 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:28,885 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:28,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:28,885 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:28,885 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:28,885 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,885 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:28,885 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:28,885 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:28,885 DEBUG [Thread-888] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:28,886 DEBUG [Thread-888] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:28,886 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:28,888 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:28,888 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:28,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:28,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:28,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:28,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:28,979 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:28,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59804 deadline: 1681034128975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,979 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:28,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59752 deadline: 1681034128975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,986 DEBUG [Thread-888] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:28,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:28,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:28,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:28,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:28,989 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:28,989 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:28,989 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:28,989 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:28,989 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:28,989 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:28,991 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,991 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:28,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:28,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:28,991 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,991 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:28,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:28,991 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:28,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:28,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:28,992 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:28,992 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:28,992 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:28,992 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:28,992 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:28,992 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:28,992 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:28,992 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:28,992 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:28,992 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. started... 2023-04-09 09:54:28,993 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. as already flushing 2023-04-09 09:54:28,993 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:28,993 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:28,993 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:28,993 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,993 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:28,993 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:28,998 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,998 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:28,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:28,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:28,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:28,999 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:28,999 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:28,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:28,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:28,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:28,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:29,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:29,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:29,000 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:29,000 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:29,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a236ae7[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:29,001 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:29,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:29,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:29,001 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:29,001 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:29,001 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:29,001 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:29,001 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:29,002 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:29,002 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:29,003 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:29,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:29,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:29,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:29,003 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:29,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:29,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:29,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:29,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:29,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:29,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:29,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:29,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:29,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:29,005 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:29,005 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:29,005 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:29,005 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:29,005 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:29,007 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:29,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:29,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:29,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:29,007 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:29,007 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:29,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:29,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:29,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:29,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:29,007 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:29,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:29,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:29,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:29,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:29,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:29,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:29,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:29,009 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:29,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:29,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:29,012 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=505, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/A/54a7c5b93b24458895ac89c44df2712d 2023-04-09 09:54:29,015 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:29,015 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:29,015 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:29,015 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:29,015 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:29,015 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:29,015 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:29,016 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:29,016 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:29,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:29,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:29,016 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:29,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:29,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:29,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:29,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:29,017 DEBUG [Thread-888] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:29,017 DEBUG [Thread-888] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:29,117 DEBUG [Thread-888] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:29,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:29,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:29,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:29,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:29,121 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:29,121 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:29,121 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:29,121 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:29,121 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:29,121 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:29,122 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:29,122 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:29,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:29,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:29,123 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:29,123 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:29,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:29,123 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:29,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:29,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:29,124 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:29,124 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:29,124 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:29,124 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:29,124 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:29,124 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:29,124 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:29,124 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:29,124 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:29,125 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. started... 2023-04-09 09:54:29,125 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. as already flushing 2023-04-09 09:54:29,125 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:29,125 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:29,125 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:29,125 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:29,125 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:29,125 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:29,126 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:29,127 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:29,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:29,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:29,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:29,127 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:29,127 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:29,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:29,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:29,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:29,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:29,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:29,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:29,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65ff2685[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:29,130 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:29,130 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:29,131 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:29,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:29,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:29,131 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:29,131 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:29,131 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:29,132 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:29,132 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:29,134 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:29,134 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:29,134 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:29,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:29,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:29,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:29,134 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:29,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:29,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:29,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:29,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:29,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:29,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:29,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:29,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:29,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:29,137 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:29,137 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:29,137 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:29,137 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:29,137 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:29,139 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:29,139 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:29,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:29,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:29,139 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:29,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:29,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:29,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:29,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:29,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:29,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:29,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:29,140 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:29,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:29,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:29,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:29,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:29,141 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:29,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:29,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:29,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:29,143 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:29,143 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:29,143 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:29,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:29,143 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:29,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:29,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:29,143 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:29,143 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:29,143 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:29,143 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:29,143 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:29,143 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:29,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:29,144 DEBUG [Thread-888] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:29,144 DEBUG [Thread-888] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:29,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:29,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:29,245 DEBUG [Thread-888] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:29,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:29,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:29,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:29,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:29,248 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:29,248 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:29,248 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:29,248 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:29,249 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:29,249 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:29,250 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:29,250 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:29,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:29,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:29,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:29,251 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:29,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:29,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:29,252 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:29,252 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:29,252 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:29,254 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:29,254 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:29,254 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:29,254 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:29,254 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:29,254 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:29,254 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:29,255 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:29,255 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. started... 2023-04-09 09:54:29,255 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. as already flushing 2023-04-09 09:54:29,255 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:29,255 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:29,255 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:29,255 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:29,255 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:29,255 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:29,258 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:29,258 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:29,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:29,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:29,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:29,258 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:29,258 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:29,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:29,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:29,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:29,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:29,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:29,260 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:29,260 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:29,260 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33c21387[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:29,260 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:29,262 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:29,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:29,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:29,262 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:29,262 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:29,262 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:29,262 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:29,262 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:29,264 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:29,264 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:29,264 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:29,264 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:29,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:29,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:29,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:29,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:29,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:29,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:29,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:29,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:29,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:29,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:29,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:29,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:29,266 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:29,266 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:29,266 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:29,266 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:29,266 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:29,268 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:29,268 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:29,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:29,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:29,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:29,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:29,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:29,268 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:29,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:29,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:29,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:29,269 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:29,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:29,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:29,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:29,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:29,270 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:29,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:29,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:29,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:29,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:29,273 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:29,273 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:29,273 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:29,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:29,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:29,273 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:29,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:29,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:29,273 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:29,273 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:29,273 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:29,273 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:29,273 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:29,273 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:29,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:29,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:29,273 DEBUG [Thread-888] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:29,273 DEBUG [Thread-888] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:29,285 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:29,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59804 deadline: 1681034129282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:29,286 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:29,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59752 deadline: 1681034129282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:29,374 DEBUG [Thread-888] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:29,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:29,376 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:29,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:29,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:29,377 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:29,377 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:29,377 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:29,377 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:29,378 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:29,378 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:29,379 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:29,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:29,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:29,379 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:29,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:29,380 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:29,380 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:29,380 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:29,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:29,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:29,381 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:29,381 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:29,381 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:29,381 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:29,381 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:29,381 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:29,384 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:29,384 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:29,384 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:29,384 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. started... 2023-04-09 09:54:29,384 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. as already flushing 2023-04-09 09:54:29,384 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:29,384 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:29,384 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:29,384 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:29,384 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:29,384 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:29,386 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:29,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:29,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:29,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:29,386 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:29,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:29,386 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:29,386 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:29,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:29,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:29,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:29,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:29,388 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:29,388 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b3f7965[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:29,388 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:29,388 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:29,395 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:29,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:29,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:29,395 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:29,395 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:29,395 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:29,395 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:29,395 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:29,396 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:29,396 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:29,397 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:29,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:29,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:29,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:29,397 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:29,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:29,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:29,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:29,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:29,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:29,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:29,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:29,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:29,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:29,399 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:29,399 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:29,399 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:29,399 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:29,399 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:29,401 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:29,401 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:29,401 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:29,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:29,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:29,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:29,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:29,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:29,401 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:29,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:29,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:29,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:29,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:29,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:29,402 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:29,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:29,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:29,405 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:29,405 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:29,405 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:29,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:29,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:29,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:29,405 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:29,405 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:29,405 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:29,405 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:29,405 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:29,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:29,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:29,405 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:29,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:29,405 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:29,406 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:29,406 DEBUG [Thread-888] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:29,406 DEBUG [Thread-888] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:29,406 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:29,435 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=505 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/B/1a5632fbb8974e6fafcc3a4b22ed1de3 2023-04-09 09:54:29,457 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=505 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/C/f56893d145d348f995e9a9fc16dc9ef5 2023-04-09 09:54:29,462 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/A/54a7c5b93b24458895ac89c44df2712d as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/54a7c5b93b24458895ac89c44df2712d 2023-04-09 09:54:29,474 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/54a7c5b93b24458895ac89c44df2712d, entries=350, sequenceid=505, filesize=52.9 K 2023-04-09 09:54:29,475 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/B/1a5632fbb8974e6fafcc3a4b22ed1de3 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/1a5632fbb8974e6fafcc3a4b22ed1de3 2023-04-09 09:54:29,479 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/1a5632fbb8974e6fafcc3a4b22ed1de3, entries=150, sequenceid=505, filesize=11.9 K 2023-04-09 09:54:29,479 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/C/f56893d145d348f995e9a9fc16dc9ef5 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/f56893d145d348f995e9a9fc16dc9ef5 2023-04-09 09:54:29,483 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/f56893d145d348f995e9a9fc16dc9ef5, entries=150, sequenceid=505, filesize=11.9 K 2023-04-09 09:54:29,484 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for b86d94f6f111f1359f84891af8d12a40 in 922ms, sequenceid=505, compaction requested=true 2023-04-09 09:54:29,484 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b86d94f6f111f1359f84891af8d12a40: 2023-04-09 09:54:29,484 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:54:29,484 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-09 09:54:29,484 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-09 09:54:29,484 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:54:29,484 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:54:29,486 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 159080 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-09 09:54:29,486 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1912): b86d94f6f111f1359f84891af8d12a40/A is initiating minor compaction (all files) 2023-04-09 09:54:29,486 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61612 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-09 09:54:29,486 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b86d94f6f111f1359f84891af8d12a40/A in TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:29,486 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): b86d94f6f111f1359f84891af8d12a40/B is initiating minor compaction (all files) 2023-04-09 09:54:29,486 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b86d94f6f111f1359f84891af8d12a40/B in TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:29,486 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/3e62ad31d3a84cce99361b410a12aaa4, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/b5e1d07360704206927465f48f02c98b, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/29284701768c44d7bec2c846c4f665cc, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/84c6441be2e446fdaf879320e18877f7, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/54a7c5b93b24458895ac89c44df2712d] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp, totalSize=155.4 K 2023-04-09 09:54:29,486 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/95e411e7f0544694b3de909e80e7f789, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/17f4fc67a0e24903832774556a38b515, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/434756787fbb49c2919d209e1d3d0bba, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/d6d30be793df4c6080a0736951d3bb0c, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/1a5632fbb8974e6fafcc3a4b22ed1de3] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp, totalSize=60.2 K 2023-04-09 09:54:29,486 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 3e62ad31d3a84cce99361b410a12aaa4, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=439, earliestPutTs=1681034062882 2023-04-09 09:54:29,487 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 95e411e7f0544694b3de909e80e7f789, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=439, earliestPutTs=1681034062882 2023-04-09 09:54:29,487 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting b5e1d07360704206927465f48f02c98b, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=459, earliestPutTs=1681034064563 2023-04-09 09:54:29,487 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 17f4fc67a0e24903832774556a38b515, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=459, earliestPutTs=1681034064563 2023-04-09 09:54:29,487 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 29284701768c44d7bec2c846c4f665cc, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=474, earliestPutTs=1681034065727 2023-04-09 09:54:29,487 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 434756787fbb49c2919d209e1d3d0bba, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=474, earliestPutTs=1681034065727 2023-04-09 09:54:29,488 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 84c6441be2e446fdaf879320e18877f7, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=493, earliestPutTs=1681034066375 2023-04-09 09:54:29,488 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting d6d30be793df4c6080a0736951d3bb0c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=493, earliestPutTs=1681034066375 2023-04-09 09:54:29,488 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 54a7c5b93b24458895ac89c44df2712d, keycount=350, bloomtype=ROW, size=52.9 K, encoding=NONE, compression=NONE, seqNum=505, earliestPutTs=1681034068524 2023-04-09 09:54:29,488 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 1a5632fbb8974e6fafcc3a4b22ed1de3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=505, earliestPutTs=1681034068549 2023-04-09 09:54:29,501 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): b86d94f6f111f1359f84891af8d12a40#B#compaction#189 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:54:29,506 DEBUG [Thread-888] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:29,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:29,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:29,513 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] throttle.PressureAwareThroughputController(145): b86d94f6f111f1359f84891af8d12a40#A#compaction#190 average throughput is 1.52 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:54:29,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:29,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:29,519 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:29,519 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:29,519 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:29,519 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:29,519 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:29,519 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:29,522 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:29,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:29,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:29,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:29,522 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:29,523 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:29,523 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:29,523 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:29,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:29,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:29,523 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:29,527 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:29,527 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:29,527 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:29,527 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:29,527 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:29,527 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:29,528 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:29,528 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:29,528 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. started... 2023-04-09 09:54:29,528 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b86d94f6f111f1359f84891af8d12a40 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-09 09:54:29,528 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=A 2023-04-09 09:54:29,529 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:54:29,529 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=B 2023-04-09 09:54:29,529 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:54:29,529 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=C 2023-04-09 09:54:29,529 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:54:29,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on b86d94f6f111f1359f84891af8d12a40 2023-04-09 09:54:29,791 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. as already flushing 2023-04-09 09:54:29,807 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:29,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59804 deadline: 1681034129800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:29,810 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:29,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59752 deadline: 1681034129807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:29,910 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:29,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59804 deadline: 1681034129908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:29,919 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:29,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59752 deadline: 1681034129911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:29,928 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/B/57b6f1f33e464e4a91e093aab09222e4 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/57b6f1f33e464e4a91e093aab09222e4 2023-04-09 09:54:29,938 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in b86d94f6f111f1359f84891af8d12a40/B of b86d94f6f111f1359f84891af8d12a40 into 57b6f1f33e464e4a91e093aab09222e4(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:54:29,938 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b86d94f6f111f1359f84891af8d12a40: 2023-04-09 09:54:29,938 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40., storeName=b86d94f6f111f1359f84891af8d12a40/B, priority=11, startTime=1681034069484; duration=0sec 2023-04-09 09:54:29,938 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:54:29,938 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-09 09:54:29,939 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61612 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-09 09:54:29,940 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): b86d94f6f111f1359f84891af8d12a40/C is initiating minor compaction (all files) 2023-04-09 09:54:29,940 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b86d94f6f111f1359f84891af8d12a40/C in TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:29,940 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/4eace68c8a0d43b984cf9d16e4257de6, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/36605c84a13449948e3a9ac065bc563e, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/d30c2449774d483392e3b42c577abc35, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/e3b74271b37f4154ae0bacf5c7924761, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/f56893d145d348f995e9a9fc16dc9ef5] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp, totalSize=60.2 K 2023-04-09 09:54:29,940 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 4eace68c8a0d43b984cf9d16e4257de6, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=439, earliestPutTs=1681034062882 2023-04-09 09:54:29,941 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 36605c84a13449948e3a9ac065bc563e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=459, earliestPutTs=1681034064563 2023-04-09 09:54:29,941 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting d30c2449774d483392e3b42c577abc35, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=474, earliestPutTs=1681034065727 2023-04-09 09:54:29,942 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting e3b74271b37f4154ae0bacf5c7924761, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=493, earliestPutTs=1681034066375 2023-04-09 09:54:29,943 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting f56893d145d348f995e9a9fc16dc9ef5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=505, earliestPutTs=1681034068549 2023-04-09 09:54:29,951 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230409c59175cdae6b4e94aea2e61f041063b8 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409c59175cdae6b4e94aea2e61f041063b8 2023-04-09 09:54:29,959 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): b86d94f6f111f1359f84891af8d12a40#C#compaction#192 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:54:29,961 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=541, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/A/705dea50a7444e8482b2d9c2e232db30 2023-04-09 09:54:29,970 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/C/e5c0b3c60f9a454d878a3c1f4c6097cc as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/e5c0b3c60f9a454d878a3c1f4c6097cc 2023-04-09 09:54:29,975 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in b86d94f6f111f1359f84891af8d12a40/C of b86d94f6f111f1359f84891af8d12a40 into e5c0b3c60f9a454d878a3c1f4c6097cc(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:54:29,976 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b86d94f6f111f1359f84891af8d12a40: 2023-04-09 09:54:29,976 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40., storeName=b86d94f6f111f1359f84891af8d12a40/C, priority=11, startTime=1681034069484; duration=0sec 2023-04-09 09:54:29,976 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:54:30,111 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:30,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59804 deadline: 1681034130111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,122 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:30,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59752 deadline: 1681034130121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,349 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/A/a7e1b582bb474b1bba512d146226ce85 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/a7e1b582bb474b1bba512d146226ce85 2023-04-09 09:54:30,357 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in b86d94f6f111f1359f84891af8d12a40/A of b86d94f6f111f1359f84891af8d12a40 into a7e1b582bb474b1bba512d146226ce85(size=26.3 K), total size for store is 26.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:54:30,357 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b86d94f6f111f1359f84891af8d12a40: 2023-04-09 09:54:30,357 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40., storeName=b86d94f6f111f1359f84891af8d12a40/A, priority=11, startTime=1681034069484; duration=0sec 2023-04-09 09:54:30,357 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:54:30,379 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=541 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/B/1a983a371fe248fb8c289bfd62b4bd56 2023-04-09 09:54:30,392 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=541 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/C/a78005267dfd4b7b9c1cd1bca8d98912 2023-04-09 09:54:30,399 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/A/705dea50a7444e8482b2d9c2e232db30 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/705dea50a7444e8482b2d9c2e232db30 2023-04-09 09:54:30,403 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/705dea50a7444e8482b2d9c2e232db30, entries=150, sequenceid=541, filesize=25.4 K 2023-04-09 09:54:30,404 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/B/1a983a371fe248fb8c289bfd62b4bd56 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/1a983a371fe248fb8c289bfd62b4bd56 2023-04-09 09:54:30,409 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/1a983a371fe248fb8c289bfd62b4bd56, entries=150, sequenceid=541, filesize=11.9 K 2023-04-09 09:54:30,410 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/C/a78005267dfd4b7b9c1cd1bca8d98912 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/a78005267dfd4b7b9c1cd1bca8d98912 2023-04-09 09:54:30,414 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/a78005267dfd4b7b9c1cd1bca8d98912, entries=150, sequenceid=541, filesize=11.9 K 2023-04-09 09:54:30,414 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for b86d94f6f111f1359f84891af8d12a40 in 886ms, sequenceid=541, compaction requested=false 2023-04-09 09:54:30,415 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b86d94f6f111f1359f84891af8d12a40: 2023-04-09 09:54:30,415 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:30,415 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:30,415 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:30,415 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,416 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:30,416 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:30,417 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,417 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:30,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:30,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:30,418 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:30,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:30,418 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:30,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:30,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:30,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on b86d94f6f111f1359f84891af8d12a40 2023-04-09 09:54:30,419 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b86d94f6f111f1359f84891af8d12a40 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-09 09:54:30,419 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=A 2023-04-09 09:54:30,419 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:30,419 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=B 2023-04-09 09:54:30,419 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:30,419 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=C 2023-04-09 09:54:30,419 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:30,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:30,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:30,426 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:30,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a7a58b7[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:30,426 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:30,427 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:30,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:30,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:30,428 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,428 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:30,428 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:30,428 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:30,428 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:30,429 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:30,429 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:30,429 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,429 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:30,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:30,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:30,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:30,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:30,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:30,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:30,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:30,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:30,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:30,433 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:30,433 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:30,433 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:30,433 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:30,433 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:30,436 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:30,436 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:30,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:30,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:30,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:30,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:30,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:30,436 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:30,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:30,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:30,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:30,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:30,439 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:30,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:30,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:30,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,440 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:30,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:30,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,444 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:30,444 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:30,444 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:30,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:30,444 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:30,444 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:30,444 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:30,444 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:30,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:30,444 DEBUG [Thread-888] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:30,444 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,445 DEBUG [Thread-888] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:30,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:30,445 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:30,445 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:30,445 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:30,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:30,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:30,455 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:30,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59752 deadline: 1681034130454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,456 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681034070440, max timestamp=1681034070454Num uniques 859; 2023-04-09 09:54:30,456 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-09 09:54:30,457 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681034070440, max timestamp=1681034070454Num uniques 859; 2023-04-09 09:54:30,457 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-09 09:54:30,457 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681034070440, max timestamp=1681034070454Num uniques 859; 2023-04-09 09:54:30,457 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-09 09:54:30,464 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304096f678fb71bf0414eb80b7dfa5582a104 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304096f678fb71bf0414eb80b7dfa5582a104 2023-04-09 09:54:30,481 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=555, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/A/a5b46aef3cdc4e62831edbfadc0dcd88 2023-04-09 09:54:30,494 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=555 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/B/e0ef3bf3a1454d7988449aed64df49c7 2023-04-09 09:54:30,500 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:30,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59804 deadline: 1681034130497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,545 DEBUG [Thread-888] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:30,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:30,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:30,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:30,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:30,548 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:30,548 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:30,548 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:30,548 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:30,548 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:30,549 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:30,550 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:30,550 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:30,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:30,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:30,550 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,550 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:30,551 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:30,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:30,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:30,551 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:30,551 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:30,551 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:30,551 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:30,551 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:30,551 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:30,551 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:30,552 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:30,552 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:30,552 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. started... 2023-04-09 09:54:30,552 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. as already flushing 2023-04-09 09:54:30,552 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:30,552 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:30,552 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:30,552 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,552 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:30,552 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:30,554 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,554 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:30,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:30,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:30,554 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:30,554 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:30,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:30,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:30,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:30,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:30,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:30,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16ad9c73[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:30,555 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:30,555 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:30,558 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:30,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:30,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:30,558 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:30,558 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,558 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:30,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59752 deadline: 1681034130557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,558 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:30,558 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:30,558 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:30,560 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,560 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:30,560 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:30,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:30,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:30,560 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:30,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:30,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:30,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:30,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:30,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:30,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:30,563 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:30,563 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:30,563 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:30,563 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:30,563 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:30,563 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:30,565 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:30,565 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:30,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:30,565 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:30,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:30,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:30,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:30,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:30,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:30,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:30,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:30,565 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:30,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:30,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:30,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:30,567 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:30,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:30,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,569 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,569 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:30,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:30,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:30,570 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:30,570 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:30,570 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:30,570 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:30,570 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,570 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:30,570 DEBUG [Thread-888] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:30,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:30,570 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:30,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:30,570 DEBUG [Thread-888] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:30,570 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:30,570 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:30,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:30,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:30,603 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:30,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59804 deadline: 1681034130602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,670 DEBUG [Thread-888] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:30,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:30,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:30,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:30,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:30,674 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:30,674 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:30,674 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:30,674 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:30,674 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:30,674 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:30,676 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,676 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:30,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:30,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:30,676 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,676 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:30,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:30,677 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:30,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:30,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:30,677 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:30,677 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:30,677 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:30,678 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:30,678 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:30,678 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:30,678 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:30,678 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:30,678 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:30,678 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. started... 2023-04-09 09:54:30,678 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. as already flushing 2023-04-09 09:54:30,678 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:30,678 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:30,678 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:30,678 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,678 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:30,678 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:30,679 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,679 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:30,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:30,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:30,680 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:30,680 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:30,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:30,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:30,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:30,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:30,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:30,681 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:30,681 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:30,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3550eb0a[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:30,682 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:30,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:30,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:30,683 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:30,683 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:30,683 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:30,683 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,683 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:30,687 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:30,687 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:30,687 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,687 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:30,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:30,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:30,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:30,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:30,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:30,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:30,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:30,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:30,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:30,690 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:30,690 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:30,690 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:30,690 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:30,690 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:30,692 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:30,692 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:30,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:30,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:30,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:30,692 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:30,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:30,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:30,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:30,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:30,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:30,692 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:30,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:30,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:30,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:30,693 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:30,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:30,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,696 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:30,696 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:30,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:30,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:30,696 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:30,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:30,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:30,696 DEBUG [Thread-888] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:30,696 DEBUG [Thread-888] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:30,697 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,698 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:30,698 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:30,698 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,698 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:30,698 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:30,698 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:30,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:30,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:30,698 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:30,762 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:30,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59752 deadline: 1681034130760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,796 DEBUG [Thread-888] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:30,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:30,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:30,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:30,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:30,800 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:30,800 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:30,800 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:30,800 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:30,801 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:30,801 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:30,802 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:30,802 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:30,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:30,802 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,802 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:30,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:30,803 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:30,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:30,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:30,803 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:30,804 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:30,804 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:30,804 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:30,804 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:30,804 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:30,804 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:30,804 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:30,804 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:30,804 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. started... 2023-04-09 09:54:30,804 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. as already flushing 2023-04-09 09:54:30,804 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:30,804 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:30,804 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:30,804 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,804 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:30,804 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:30,805 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:30,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59804 deadline: 1681034130804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,806 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,806 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:30,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:30,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:30,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:30,807 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:30,807 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:30,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:30,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:30,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:30,808 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:30,808 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75645894[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:30,808 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:30,808 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:30,810 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:30,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:30,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:30,810 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:30,810 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:30,810 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:30,810 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,810 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:30,812 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,812 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:30,812 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:30,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:30,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:30,812 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:30,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:30,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:30,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:30,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:30,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:30,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:30,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:30,814 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:30,815 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:30,815 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:30,815 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:30,815 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:30,817 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:30,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:30,817 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:30,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:30,817 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:30,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:30,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:30,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:30,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:30,818 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:30,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:30,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:30,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:30,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:30,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:30,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,819 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:30,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:30,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,822 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,822 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:30,823 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:30,823 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:30,823 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:30,823 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:30,823 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:30,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:30,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:30,823 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:30,823 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:30,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:30,823 DEBUG [Thread-888] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:30,823 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:30,823 DEBUG [Thread-888] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:30,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:30,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:30,906 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=555 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/C/1e4a3653b11d4297acabcf663054c80f 2023-04-09 09:54:30,910 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/A/a5b46aef3cdc4e62831edbfadc0dcd88 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/a5b46aef3cdc4e62831edbfadc0dcd88 2023-04-09 09:54:30,915 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/a5b46aef3cdc4e62831edbfadc0dcd88, entries=200, sequenceid=555, filesize=32.3 K 2023-04-09 09:54:30,917 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/B/e0ef3bf3a1454d7988449aed64df49c7 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/e0ef3bf3a1454d7988449aed64df49c7 2023-04-09 09:54:30,921 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/e0ef3bf3a1454d7988449aed64df49c7, entries=150, sequenceid=555, filesize=11.9 K 2023-04-09 09:54:30,922 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/C/1e4a3653b11d4297acabcf663054c80f as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/1e4a3653b11d4297acabcf663054c80f 2023-04-09 09:54:30,923 DEBUG [Thread-888] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:30,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:30,926 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:30,926 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/1e4a3653b11d4297acabcf663054c80f, entries=150, sequenceid=555, filesize=11.9 K 2023-04-09 09:54:30,927 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for b86d94f6f111f1359f84891af8d12a40 in 508ms, sequenceid=555, compaction requested=true 2023-04-09 09:54:30,927 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b86d94f6f111f1359f84891af8d12a40: 2023-04-09 09:54:30,928 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:54:30,928 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:54:30,928 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-09 09:54:30,928 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-09 09:54:30,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:30,928 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:54:30,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:30,928 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:30,928 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:30,928 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:30,928 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:30,928 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:30,929 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:30,930 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:30,930 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:30,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:30,930 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:30,930 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:30,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:30,931 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:30,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:30,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:30,931 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:30,931 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:30,931 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:30,932 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:30,932 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:30,932 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:30,932 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-09 09:54:30,932 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:30,932 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:30,932 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 86030 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-09 09:54:30,932 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1912): b86d94f6f111f1359f84891af8d12a40/A is initiating minor compaction (all files) 2023-04-09 09:54:30,932 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b86d94f6f111f1359f84891af8d12a40/A in TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:30,932 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/a7e1b582bb474b1bba512d146226ce85, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/705dea50a7444e8482b2d9c2e232db30, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/a5b46aef3cdc4e62831edbfadc0dcd88] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp, totalSize=84.0 K 2023-04-09 09:54:30,932 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): b86d94f6f111f1359f84891af8d12a40/B is initiating minor compaction (all files) 2023-04-09 09:54:30,932 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b86d94f6f111f1359f84891af8d12a40/B in TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:30,933 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/57b6f1f33e464e4a91e093aab09222e4, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/1a983a371fe248fb8c289bfd62b4bd56, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/e0ef3bf3a1454d7988449aed64df49c7] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp, totalSize=36.5 K 2023-04-09 09:54:30,932 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:30,933 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. started... 2023-04-09 09:54:30,933 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b86d94f6f111f1359f84891af8d12a40 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-04-09 09:54:30,933 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting a7e1b582bb474b1bba512d146226ce85, keycount=150, bloomtype=ROW, size=26.3 K, encoding=NONE, compression=NONE, seqNum=505, earliestPutTs=1681034068549 2023-04-09 09:54:30,933 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=A 2023-04-09 09:54:30,933 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:54:30,933 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 57b6f1f33e464e4a91e093aab09222e4, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=505, earliestPutTs=1681034068549 2023-04-09 09:54:30,933 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=B 2023-04-09 09:54:30,934 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:54:30,934 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 1a983a371fe248fb8c289bfd62b4bd56, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=541, earliestPutTs=1681034068651 2023-04-09 09:54:30,934 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 705dea50a7444e8482b2d9c2e232db30, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=541, earliestPutTs=1681034068651 2023-04-09 09:54:30,934 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=C 2023-04-09 09:54:30,934 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:54:30,934 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting e0ef3bf3a1454d7988449aed64df49c7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=555, earliestPutTs=1681034069794 2023-04-09 09:54:30,934 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting a5b46aef3cdc4e62831edbfadc0dcd88, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=555, earliestPutTs=1681034069794 2023-04-09 09:54:30,943 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): b86d94f6f111f1359f84891af8d12a40#B#compaction#200 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:54:30,959 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] throttle.PressureAwareThroughputController(145): b86d94f6f111f1359f84891af8d12a40#A#compaction#199 average throughput is 1.16 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:54:31,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on b86d94f6f111f1359f84891af8d12a40 2023-04-09 09:54:31,068 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. as already flushing 2023-04-09 09:54:31,089 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:31,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59788 deadline: 1681034131086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:31,094 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:31,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59752 deadline: 1681034131089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:31,109 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:31,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59804 deadline: 1681034131107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:31,194 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:31,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59788 deadline: 1681034131190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:31,198 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:31,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59752 deadline: 1681034131196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:31,361 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304096171f39ec73f40568f1950e228f35834 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304096171f39ec73f40568f1950e228f35834 2023-04-09 09:54:31,368 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/B/a0ad05fc6a124df8aab5ee7edf4f1c1f as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/a0ad05fc6a124df8aab5ee7edf4f1c1f 2023-04-09 09:54:31,369 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=589, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/A/48ac17de479a4fbfa6464c43a13edd16 2023-04-09 09:54:31,377 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b86d94f6f111f1359f84891af8d12a40/B of b86d94f6f111f1359f84891af8d12a40 into a0ad05fc6a124df8aab5ee7edf4f1c1f(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:54:31,377 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b86d94f6f111f1359f84891af8d12a40: 2023-04-09 09:54:31,377 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40., storeName=b86d94f6f111f1359f84891af8d12a40/B, priority=13, startTime=1681034070928; duration=0sec 2023-04-09 09:54:31,378 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:54:31,378 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-09 09:54:31,380 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-09 09:54:31,380 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): b86d94f6f111f1359f84891af8d12a40/C is initiating minor compaction (all files) 2023-04-09 09:54:31,382 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b86d94f6f111f1359f84891af8d12a40/C in TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:31,382 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/e5c0b3c60f9a454d878a3c1f4c6097cc, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/a78005267dfd4b7b9c1cd1bca8d98912, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/1e4a3653b11d4297acabcf663054c80f] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp, totalSize=36.5 K 2023-04-09 09:54:31,385 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting e5c0b3c60f9a454d878a3c1f4c6097cc, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=505, earliestPutTs=1681034068549 2023-04-09 09:54:31,385 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting a78005267dfd4b7b9c1cd1bca8d98912, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=541, earliestPutTs=1681034068651 2023-04-09 09:54:31,386 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 1e4a3653b11d4297acabcf663054c80f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=555, earliestPutTs=1681034069794 2023-04-09 09:54:31,399 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:31,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59788 deadline: 1681034131397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:31,403 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:31,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59752 deadline: 1681034131400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:31,417 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): b86d94f6f111f1359f84891af8d12a40#C#compaction#202 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:54:31,437 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/A/8aeb41580f44426d9d93867f35243ad0 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/8aeb41580f44426d9d93867f35243ad0 2023-04-09 09:54:31,444 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b86d94f6f111f1359f84891af8d12a40/A of b86d94f6f111f1359f84891af8d12a40 into 8aeb41580f44426d9d93867f35243ad0(size=26.4 K), total size for store is 26.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:54:31,444 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b86d94f6f111f1359f84891af8d12a40: 2023-04-09 09:54:31,444 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40., storeName=b86d94f6f111f1359f84891af8d12a40/A, priority=13, startTime=1681034070928; duration=0sec 2023-04-09 09:54:31,444 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:54:31,445 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/C/ac60ecdb5f394909a93090d48f96627e as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/ac60ecdb5f394909a93090d48f96627e 2023-04-09 09:54:31,450 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b86d94f6f111f1359f84891af8d12a40/C of b86d94f6f111f1359f84891af8d12a40 into ac60ecdb5f394909a93090d48f96627e(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:54:31,450 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b86d94f6f111f1359f84891af8d12a40: 2023-04-09 09:54:31,450 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40., storeName=b86d94f6f111f1359f84891af8d12a40/C, priority=13, startTime=1681034070928; duration=0sec 2023-04-09 09:54:31,450 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:54:31,643 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:31,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59804 deadline: 1681034131639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:31,700 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:31,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59788 deadline: 1681034131700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:31,707 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:31,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59752 deadline: 1681034131706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:31,830 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=589 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/B/d06ea1455fcf4594acf1567e20d0ef31 2023-04-09 09:54:31,844 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=589 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/C/f7816e07b14a40d2a452c96aed6a4e7c 2023-04-09 09:54:31,850 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/A/48ac17de479a4fbfa6464c43a13edd16 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/48ac17de479a4fbfa6464c43a13edd16 2023-04-09 09:54:31,854 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/48ac17de479a4fbfa6464c43a13edd16, entries=150, sequenceid=589, filesize=25.4 K 2023-04-09 09:54:31,855 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/B/d06ea1455fcf4594acf1567e20d0ef31 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/d06ea1455fcf4594acf1567e20d0ef31 2023-04-09 09:54:31,867 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/d06ea1455fcf4594acf1567e20d0ef31, entries=150, sequenceid=589, filesize=11.9 K 2023-04-09 09:54:31,868 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/C/f7816e07b14a40d2a452c96aed6a4e7c as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/f7816e07b14a40d2a452c96aed6a4e7c 2023-04-09 09:54:31,886 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/f7816e07b14a40d2a452c96aed6a4e7c, entries=150, sequenceid=589, filesize=11.9 K 2023-04-09 09:54:31,887 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=53.67 KB/54960 for b86d94f6f111f1359f84891af8d12a40 in 954ms, sequenceid=589, compaction requested=false 2023-04-09 09:54:31,887 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b86d94f6f111f1359f84891af8d12a40: 2023-04-09 09:54:31,887 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:31,887 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:31,887 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:31,887 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:31,887 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:31,887 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:31,891 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:31,891 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:31,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:31,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:31,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:31,892 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:31,892 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:31,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:31,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:31,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:31,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:31,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:31,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:31,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d03096f[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:31,893 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:31,893 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:31,895 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:31,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:31,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:31,895 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:31,895 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:31,895 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:31,895 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:31,895 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:31,897 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:31,897 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:31,897 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:31,897 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:31,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:31,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:31,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:31,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:31,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:31,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:31,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:31,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:31,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:31,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:31,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:31,900 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:31,900 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:31,900 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:31,900 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:31,900 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:31,900 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:31,902 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:31,902 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:31,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:31,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:31,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:31,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:31,903 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:31,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:31,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:31,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:31,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:31,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:31,903 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:31,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:31,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:31,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:31,905 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:31,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:31,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:31,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:31,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:31,908 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:31,908 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:31,908 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:31,908 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:31,908 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:31,908 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:31,908 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:31,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:31,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:31,908 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:31,908 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:31,908 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:31,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:31,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:31,908 DEBUG [Thread-888] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:31,908 DEBUG [Thread-888] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:31,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:31,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:31,909 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:32,009 DEBUG [Thread-888] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:32,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:32,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:32,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:32,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:32,028 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:32,028 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:32,028 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:32,028 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:32,028 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:32,028 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:32,030 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:32,030 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:32,030 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:32,030 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:32,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:32,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:32,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:32,031 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:32,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:32,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:32,031 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:32,031 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:32,031 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:32,032 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:32,032 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:32,032 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:32,032 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:32,032 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:32,032 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:32,032 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. started... 2023-04-09 09:54:32,032 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b86d94f6f111f1359f84891af8d12a40 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-09 09:54:32,033 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=A 2023-04-09 09:54:32,033 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:32,033 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=B 2023-04-09 09:54:32,033 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:32,033 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=C 2023-04-09 09:54:32,033 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:32,067 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304094c948caa14a1466c94e09bb13635f1ba to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304094c948caa14a1466c94e09bb13635f1ba 2023-04-09 09:54:32,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on b86d94f6f111f1359f84891af8d12a40 2023-04-09 09:54:32,207 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. as already flushing 2023-04-09 09:54:32,230 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034072222, max timestamp=1681034072228Num uniques 859; 2023-04-09 09:54:32,230 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:54:32,230 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034072222, max timestamp=1681034072228Num uniques 859; 2023-04-09 09:54:32,230 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034072222, max timestamp=1681034072228Num uniques 859; 2023-04-09 09:54:32,230 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:54:32,230 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:54:32,255 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:32,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59752 deadline: 1681034132253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:32,256 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:32,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59788 deadline: 1681034132254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:32,358 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:32,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59752 deadline: 1681034132357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:32,358 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:32,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59788 deadline: 1681034132357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:32,476 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=603, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/A/1981c245f3e54c74b78643fae50e5fc4 2023-04-09 09:54:32,561 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:32,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59788 deadline: 1681034132560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:32,561 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:32,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59752 deadline: 1681034132561, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:32,655 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:32,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59804 deadline: 1681034132653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:32,867 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:32,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59788 deadline: 1681034132864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:32,867 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:32,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59752 deadline: 1681034132864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:32,887 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=603 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/B/72ef84f31b4944b5929544771c568edf 2023-04-09 09:54:32,936 DEBUG [Thread-891] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x498694cb to 127.0.0.1:57816 2023-04-09 09:54:32,936 DEBUG [Thread-891] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-09 09:54:32,937 DEBUG [Thread-889] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5c472cfd to 127.0.0.1:57816 2023-04-09 09:54:32,937 DEBUG [Thread-889] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-09 09:54:32,938 DEBUG [Thread-893] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x421e5356 to 127.0.0.1:57816 2023-04-09 09:54:32,938 DEBUG [Thread-893] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-09 09:54:32,943 DEBUG [Thread-895] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4a0fd7b4 to 127.0.0.1:57816 2023-04-09 09:54:32,943 DEBUG [Thread-895] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-09 09:54:33,298 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=603 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/C/4832054364cf41ce875095d57fee191e 2023-04-09 09:54:33,302 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/A/1981c245f3e54c74b78643fae50e5fc4 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/1981c245f3e54c74b78643fae50e5fc4 2023-04-09 09:54:33,305 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/1981c245f3e54c74b78643fae50e5fc4, entries=150, sequenceid=603, filesize=25.4 K 2023-04-09 09:54:33,305 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/B/72ef84f31b4944b5929544771c568edf as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/72ef84f31b4944b5929544771c568edf 2023-04-09 09:54:33,309 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/72ef84f31b4944b5929544771c568edf, entries=150, sequenceid=603, filesize=11.9 K 2023-04-09 09:54:33,309 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/C/4832054364cf41ce875095d57fee191e as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/4832054364cf41ce875095d57fee191e 2023-04-09 09:54:33,312 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/4832054364cf41ce875095d57fee191e, entries=150, sequenceid=603, filesize=11.9 K 2023-04-09 09:54:33,313 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for b86d94f6f111f1359f84891af8d12a40 in 1281ms, sequenceid=603, compaction requested=true 2023-04-09 09:54:33,313 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b86d94f6f111f1359f84891af8d12a40: 2023-04-09 09:54:33,313 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:33,313 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:33,313 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:33,313 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:33,313 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:33,313 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:33,316 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:33,316 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:33,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:33,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:33,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:33,316 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:33,316 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:33,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:33,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:33,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:33,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:33,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:33,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:33,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@573756b8[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:33,317 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:33,318 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:33,319 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:33,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:33,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:33,319 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:33,319 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:33,319 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:33,319 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:33,319 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:33,322 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:33,322 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:33,322 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:33,322 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:33,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:33,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:33,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:33,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:33,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:33,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:33,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:33,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:33,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:33,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:33,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:33,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:33,324 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:33,324 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:33,324 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:33,324 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:33,324 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:33,330 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:33,330 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:33,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:33,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:33,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:33,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:33,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:33,330 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:33,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:33,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:33,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:33,330 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:33,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:33,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:33,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:33,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:33,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:33,331 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:33,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:33,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:33,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:33,333 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:33,333 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:33,333 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:33,333 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:33,333 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:33,333 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:33,333 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:33,333 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:33,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:33,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:33,333 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:33,333 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:33,334 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:33,334 DEBUG [Thread-888] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:33,334 DEBUG [Thread-888] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:33,334 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:33,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:33,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:33,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on b86d94f6f111f1359f84891af8d12a40 2023-04-09 09:54:33,372 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b86d94f6f111f1359f84891af8d12a40 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-09 09:54:33,372 DEBUG [Thread-886] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x370947cb to 127.0.0.1:57816 2023-04-09 09:54:33,372 DEBUG [Thread-886] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-09 09:54:33,372 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=A 2023-04-09 09:54:33,372 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:54:33,372 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=B 2023-04-09 09:54:33,372 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:54:33,372 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=C 2023-04-09 09:54:33,372 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:54:33,373 DEBUG [Thread-884] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x73e51571 to 127.0.0.1:57816 2023-04-09 09:54:33,373 DEBUG [Thread-884] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-09 09:54:33,434 DEBUG [Thread-888] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:33,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:33,786 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230409ad9f95631a5248b381fa95650a74c289 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409ad9f95631a5248b381fa95650a74c289 2023-04-09 09:54:34,191 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=640, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/A/f6155267d224449dbf998d082f535503 2023-04-09 09:54:34,202 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=640 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/B/8c5336167fc74dc3a007f54479d0381d 2023-04-09 09:54:34,613 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=640 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/C/786961940bb449bfae54ebce35d9b538 2023-04-09 09:54:34,617 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/A/f6155267d224449dbf998d082f535503 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/f6155267d224449dbf998d082f535503 2023-04-09 09:54:34,620 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/f6155267d224449dbf998d082f535503, entries=150, sequenceid=640, filesize=25.4 K 2023-04-09 09:54:34,621 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/B/8c5336167fc74dc3a007f54479d0381d as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/8c5336167fc74dc3a007f54479d0381d 2023-04-09 09:54:34,624 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/8c5336167fc74dc3a007f54479d0381d, entries=150, sequenceid=640, filesize=11.9 K 2023-04-09 09:54:34,625 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/C/786961940bb449bfae54ebce35d9b538 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/786961940bb449bfae54ebce35d9b538 2023-04-09 09:54:34,634 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/786961940bb449bfae54ebce35d9b538, entries=150, sequenceid=640, filesize=11.9 K 2023-04-09 09:54:34,634 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=6.71 KB/6870 for b86d94f6f111f1359f84891af8d12a40 in 1263ms, sequenceid=640, compaction requested=true 2023-04-09 09:54:34,634 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b86d94f6f111f1359f84891af8d12a40: 2023-04-09 09:54:34,635 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:54:34,635 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-09 09:54:34,635 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:54:34,635 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-09 09:54:34,635 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:54:34,636 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 105138 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-09 09:54:34,636 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49698 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-09 09:54:34,636 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1912): b86d94f6f111f1359f84891af8d12a40/A is initiating minor compaction (all files) 2023-04-09 09:54:34,637 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): b86d94f6f111f1359f84891af8d12a40/B is initiating minor compaction (all files) 2023-04-09 09:54:34,637 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b86d94f6f111f1359f84891af8d12a40/A in TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:34,637 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b86d94f6f111f1359f84891af8d12a40/B in TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:34,637 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/8aeb41580f44426d9d93867f35243ad0, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/48ac17de479a4fbfa6464c43a13edd16, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/1981c245f3e54c74b78643fae50e5fc4, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/f6155267d224449dbf998d082f535503] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp, totalSize=102.7 K 2023-04-09 09:54:34,637 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/a0ad05fc6a124df8aab5ee7edf4f1c1f, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/d06ea1455fcf4594acf1567e20d0ef31, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/72ef84f31b4944b5929544771c568edf, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/8c5336167fc74dc3a007f54479d0381d] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp, totalSize=48.5 K 2023-04-09 09:54:34,637 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 8aeb41580f44426d9d93867f35243ad0, keycount=150, bloomtype=ROW, size=26.4 K, encoding=NONE, compression=NONE, seqNum=555, earliestPutTs=1681034069794 2023-04-09 09:54:34,637 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting a0ad05fc6a124df8aab5ee7edf4f1c1f, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=555, earliestPutTs=1681034069794 2023-04-09 09:54:34,638 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 48ac17de479a4fbfa6464c43a13edd16, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=589, earliestPutTs=1681034070461 2023-04-09 09:54:34,638 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting d06ea1455fcf4594acf1567e20d0ef31, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=589, earliestPutTs=1681034070461 2023-04-09 09:54:34,638 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 1981c245f3e54c74b78643fae50e5fc4, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=603, earliestPutTs=1681034071079 2023-04-09 09:54:34,638 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 72ef84f31b4944b5929544771c568edf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=603, earliestPutTs=1681034071079 2023-04-09 09:54:34,638 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting f6155267d224449dbf998d082f535503, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=640, earliestPutTs=1681034072245 2023-04-09 09:54:34,638 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 8c5336167fc74dc3a007f54479d0381d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=640, earliestPutTs=1681034072245 2023-04-09 09:54:34,655 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): b86d94f6f111f1359f84891af8d12a40#B#compaction#211 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:54:34,656 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] throttle.PressureAwareThroughputController(145): b86d94f6f111f1359f84891af8d12a40#A#compaction#210 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:54:34,665 DEBUG [Thread-882] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x02bab6b2 to 127.0.0.1:57816 2023-04-09 09:54:34,665 DEBUG [Thread-882] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-09 09:54:35,064 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/B/22a462194b514f10b9838bea42c54aaf as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/22a462194b514f10b9838bea42c54aaf 2023-04-09 09:54:35,070 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b86d94f6f111f1359f84891af8d12a40/B of b86d94f6f111f1359f84891af8d12a40 into 22a462194b514f10b9838bea42c54aaf(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:54:35,070 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b86d94f6f111f1359f84891af8d12a40: 2023-04-09 09:54:35,070 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40., storeName=b86d94f6f111f1359f84891af8d12a40/B, priority=12, startTime=1681034074635; duration=0sec 2023-04-09 09:54:35,070 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:54:35,070 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-09 09:54:35,071 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49698 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-09 09:54:35,071 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): b86d94f6f111f1359f84891af8d12a40/C is initiating minor compaction (all files) 2023-04-09 09:54:35,071 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b86d94f6f111f1359f84891af8d12a40/C in TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:35,071 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/ac60ecdb5f394909a93090d48f96627e, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/f7816e07b14a40d2a452c96aed6a4e7c, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/4832054364cf41ce875095d57fee191e, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/786961940bb449bfae54ebce35d9b538] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp, totalSize=48.5 K 2023-04-09 09:54:35,071 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting ac60ecdb5f394909a93090d48f96627e, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=555, earliestPutTs=1681034069794 2023-04-09 09:54:35,072 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting f7816e07b14a40d2a452c96aed6a4e7c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=589, earliestPutTs=1681034070461 2023-04-09 09:54:35,072 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/A/fa83889a6b5f44ac8b3f2cfa2d0ef17b as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/fa83889a6b5f44ac8b3f2cfa2d0ef17b 2023-04-09 09:54:35,072 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 4832054364cf41ce875095d57fee191e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=603, earliestPutTs=1681034071079 2023-04-09 09:54:35,072 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 786961940bb449bfae54ebce35d9b538, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=640, earliestPutTs=1681034072245 2023-04-09 09:54:35,075 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b86d94f6f111f1359f84891af8d12a40/A of b86d94f6f111f1359f84891af8d12a40 into fa83889a6b5f44ac8b3f2cfa2d0ef17b(size=26.5 K), total size for store is 26.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:54:35,076 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b86d94f6f111f1359f84891af8d12a40: 2023-04-09 09:54:35,076 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40., storeName=b86d94f6f111f1359f84891af8d12a40/A, priority=12, startTime=1681034074634; duration=0sec 2023-04-09 09:54:35,076 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:54:35,079 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): b86d94f6f111f1359f84891af8d12a40#C#compaction#212 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:54:35,488 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/C/d68fc25e18a349c0aea09fdd47e2888b as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/d68fc25e18a349c0aea09fdd47e2888b 2023-04-09 09:54:35,493 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b86d94f6f111f1359f84891af8d12a40/C of b86d94f6f111f1359f84891af8d12a40 into d68fc25e18a349c0aea09fdd47e2888b(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:54:35,493 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b86d94f6f111f1359f84891af8d12a40: 2023-04-09 09:54:35,493 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40., storeName=b86d94f6f111f1359f84891af8d12a40/C, priority=12, startTime=1681034074635; duration=0sec 2023-04-09 09:54:35,493 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:54:36,540 DEBUG [Thread-878] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x29f78b2c to 127.0.0.1:57816 2023-04-09 09:54:36,540 DEBUG [Thread-878] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-09 09:54:36,547 DEBUG [Thread-880] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7bbdba17 to 127.0.0.1:57816 2023-04-09 09:54:36,547 DEBUG [Thread-880] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-09 09:54:36,547 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-09 09:54:36,548 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(394): wrote 59 2023-04-09 09:54:36,548 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(394): wrote 82 2023-04-09 09:54:36,548 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(394): wrote 108 2023-04-09 09:54:36,548 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(394): wrote 138 2023-04-09 09:54:36,548 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(394): wrote 155 2023-04-09 09:54:36,548 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-09 09:54:36,548 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(398): read 5113 2023-04-09 09:54:36,548 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(398): read 5047 2023-04-09 09:54:36,548 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-09 09:54:36,548 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(402): scanned 2181 2023-04-09 09:54:36,548 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(403): verified 6540 rows 2023-04-09 09:54:36,548 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(402): scanned 2171 2023-04-09 09:54:36,548 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(403): verified 6513 rows 2023-04-09 09:54:36,548 INFO [Listener at localhost/44563] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-09 09:54:36,548 DEBUG [Listener at localhost/44563] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x31436cf6 to 127.0.0.1:57816 2023-04-09 09:54:36,548 DEBUG [Listener at localhost/44563] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-09 09:54:36,550 INFO [Listener at localhost/44563] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-09 09:54:36,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-09 09:54:36,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-09 09:54:36,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-09 09:54:36,556 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681034076556"}]},"ts":"1681034076556"} 2023-04-09 09:54:36,557 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-09 09:54:36,560 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-09 09:54:36,560 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=b86d94f6f111f1359f84891af8d12a40, UNASSIGN}] 2023-04-09 09:54:36,562 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=b86d94f6f111f1359f84891af8d12a40, UNASSIGN 2023-04-09 09:54:36,562 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=b86d94f6f111f1359f84891af8d12a40, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:36,562 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681034076562"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681034076562"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681034076562"}]},"ts":"1681034076562"} 2023-04-09 09:54:36,563 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; CloseRegionProcedure b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544}] 2023-04-09 09:54:36,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-09 09:54:36,714 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close b86d94f6f111f1359f84891af8d12a40 2023-04-09 09:54:36,714 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing b86d94f6f111f1359f84891af8d12a40, disabling compactions & flushes 2023-04-09 09:54:36,714 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:36,714 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:36,714 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. after waiting 0 ms 2023-04-09 09:54:36,714 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:36,714 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing b86d94f6f111f1359f84891af8d12a40 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-09 09:54:36,715 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=A 2023-04-09 09:54:36,715 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:36,715 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=B 2023-04-09 09:54:36,715 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:36,715 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b86d94f6f111f1359f84891af8d12a40, store=C 2023-04-09 09:54:36,715 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:36,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-09 09:54:37,126 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230409b8f0cff7a1a94e479f3d0454c72c0db9 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409b8f0cff7a1a94e479f3d0454c72c0db9 2023-04-09 09:54:37,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-09 09:54:37,531 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=650, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/A/6f8dfa163b04410d85b9493f2261a20d 2023-04-09 09:54:37,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-09 09:54:37,941 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=650 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/B/4095f8f778554cfc8317e4612e90f841 2023-04-09 09:54:38,351 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=650 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/C/8215b6f2328d40baa754ffb3cdede059 2023-04-09 09:54:38,357 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/A/6f8dfa163b04410d85b9493f2261a20d as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/6f8dfa163b04410d85b9493f2261a20d 2023-04-09 09:54:38,362 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/6f8dfa163b04410d85b9493f2261a20d, entries=100, sequenceid=650, filesize=18.6 K 2023-04-09 09:54:38,363 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/B/4095f8f778554cfc8317e4612e90f841 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/4095f8f778554cfc8317e4612e90f841 2023-04-09 09:54:38,370 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/4095f8f778554cfc8317e4612e90f841, entries=100, sequenceid=650, filesize=9.5 K 2023-04-09 09:54:38,371 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/.tmp/C/8215b6f2328d40baa754ffb3cdede059 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/8215b6f2328d40baa754ffb3cdede059 2023-04-09 09:54:38,378 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/8215b6f2328d40baa754ffb3cdede059, entries=100, sequenceid=650, filesize=9.5 K 2023-04-09 09:54:38,379 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for b86d94f6f111f1359f84891af8d12a40 in 1665ms, sequenceid=650, compaction requested=false 2023-04-09 09:54:38,409 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/197873c5abc04c129b4e40349c8944c5, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/e5ea7c3ce89240e99a328e3ed77eadbe, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/6165fcd6cc284b26838e6a027e142680, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/3359657996a64dd8b74c07c96c1c33d6, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/95894e58446548c1b168ffa29667e33c, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/00795246b54341b0a7418e8de32740b2, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/25712ce6b28640e782c713352b73f1d9, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/cf072f3660e1490087a0523afd623ff8, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/d5016f012a35410da616ad1c87b7a425, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/982777c39d494d11b27e4f9784f1ab9d, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/1f732d5ef21c462191ca469b42c5f482, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/2977c3dba29347f5b7c410a14778065d, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/d876dbb86989410a839bfd16fb7c24e3, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/5cc3a69de617432e81080d9f250c12c5, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/7330f05ddf164e649d24a063e1bb5fe3, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/d4212a57fb5b472aaa6bea09d200b4b4, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/a85aea19f47a4ac3ba66ea4e58a01967, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/deec1cf53c2c478d849a356122f3ee70, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/349bad1c4d7c49fda3a539991da3786d, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/3e62ad31d3a84cce99361b410a12aaa4, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/3197f69ce7934090a7dfb01887e038dd, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/b5e1d07360704206927465f48f02c98b, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/29284701768c44d7bec2c846c4f665cc, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/84c6441be2e446fdaf879320e18877f7, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/54a7c5b93b24458895ac89c44df2712d, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/a7e1b582bb474b1bba512d146226ce85, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/705dea50a7444e8482b2d9c2e232db30, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/a5b46aef3cdc4e62831edbfadc0dcd88, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/8aeb41580f44426d9d93867f35243ad0, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/48ac17de479a4fbfa6464c43a13edd16, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/1981c245f3e54c74b78643fae50e5fc4, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/f6155267d224449dbf998d082f535503] to archive 2023-04-09 09:54:38,410 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-09 09:54:38,412 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/197873c5abc04c129b4e40349c8944c5 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/197873c5abc04c129b4e40349c8944c5 2023-04-09 09:54:38,413 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/e5ea7c3ce89240e99a328e3ed77eadbe to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/e5ea7c3ce89240e99a328e3ed77eadbe 2023-04-09 09:54:38,414 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/6165fcd6cc284b26838e6a027e142680 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/6165fcd6cc284b26838e6a027e142680 2023-04-09 09:54:38,415 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/3359657996a64dd8b74c07c96c1c33d6 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/3359657996a64dd8b74c07c96c1c33d6 2023-04-09 09:54:38,417 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/95894e58446548c1b168ffa29667e33c to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/95894e58446548c1b168ffa29667e33c 2023-04-09 09:54:38,418 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/00795246b54341b0a7418e8de32740b2 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/00795246b54341b0a7418e8de32740b2 2023-04-09 09:54:38,419 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/25712ce6b28640e782c713352b73f1d9 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/25712ce6b28640e782c713352b73f1d9 2023-04-09 09:54:38,420 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/cf072f3660e1490087a0523afd623ff8 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/cf072f3660e1490087a0523afd623ff8 2023-04-09 09:54:38,421 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/d5016f012a35410da616ad1c87b7a425 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/d5016f012a35410da616ad1c87b7a425 2023-04-09 09:54:38,424 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/982777c39d494d11b27e4f9784f1ab9d to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/982777c39d494d11b27e4f9784f1ab9d 2023-04-09 09:54:38,425 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/1f732d5ef21c462191ca469b42c5f482 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/1f732d5ef21c462191ca469b42c5f482 2023-04-09 09:54:38,427 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/2977c3dba29347f5b7c410a14778065d to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/2977c3dba29347f5b7c410a14778065d 2023-04-09 09:54:38,428 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/d876dbb86989410a839bfd16fb7c24e3 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/d876dbb86989410a839bfd16fb7c24e3 2023-04-09 09:54:38,430 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/5cc3a69de617432e81080d9f250c12c5 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/5cc3a69de617432e81080d9f250c12c5 2023-04-09 09:54:38,432 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/7330f05ddf164e649d24a063e1bb5fe3 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/7330f05ddf164e649d24a063e1bb5fe3 2023-04-09 09:54:38,433 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/d4212a57fb5b472aaa6bea09d200b4b4 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/d4212a57fb5b472aaa6bea09d200b4b4 2023-04-09 09:54:38,435 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/a85aea19f47a4ac3ba66ea4e58a01967 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/a85aea19f47a4ac3ba66ea4e58a01967 2023-04-09 09:54:38,436 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/deec1cf53c2c478d849a356122f3ee70 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/deec1cf53c2c478d849a356122f3ee70 2023-04-09 09:54:38,437 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/349bad1c4d7c49fda3a539991da3786d to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/349bad1c4d7c49fda3a539991da3786d 2023-04-09 09:54:38,439 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/3e62ad31d3a84cce99361b410a12aaa4 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/3e62ad31d3a84cce99361b410a12aaa4 2023-04-09 09:54:38,440 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/3197f69ce7934090a7dfb01887e038dd to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/3197f69ce7934090a7dfb01887e038dd 2023-04-09 09:54:38,441 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/b5e1d07360704206927465f48f02c98b to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/b5e1d07360704206927465f48f02c98b 2023-04-09 09:54:38,442 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/29284701768c44d7bec2c846c4f665cc to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/29284701768c44d7bec2c846c4f665cc 2023-04-09 09:54:38,444 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/84c6441be2e446fdaf879320e18877f7 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/84c6441be2e446fdaf879320e18877f7 2023-04-09 09:54:38,445 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/54a7c5b93b24458895ac89c44df2712d to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/54a7c5b93b24458895ac89c44df2712d 2023-04-09 09:54:38,446 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/a7e1b582bb474b1bba512d146226ce85 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/a7e1b582bb474b1bba512d146226ce85 2023-04-09 09:54:38,447 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/705dea50a7444e8482b2d9c2e232db30 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/705dea50a7444e8482b2d9c2e232db30 2023-04-09 09:54:38,449 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/a5b46aef3cdc4e62831edbfadc0dcd88 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/a5b46aef3cdc4e62831edbfadc0dcd88 2023-04-09 09:54:38,450 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/8aeb41580f44426d9d93867f35243ad0 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/8aeb41580f44426d9d93867f35243ad0 2023-04-09 09:54:38,452 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/48ac17de479a4fbfa6464c43a13edd16 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/48ac17de479a4fbfa6464c43a13edd16 2023-04-09 09:54:38,453 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/1981c245f3e54c74b78643fae50e5fc4 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/1981c245f3e54c74b78643fae50e5fc4 2023-04-09 09:54:38,454 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/f6155267d224449dbf998d082f535503 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/f6155267d224449dbf998d082f535503 2023-04-09 09:54:38,457 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/4f639fd99d8f423c8021f57d15986701, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/a19da86c8d2b4922a3309a1d5bceb826, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/15f2f6b0d9f94eaabb88ac1b4c29d88e, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/5af34ea9f9e04bf3b298b87dec4719ad, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/6c8ea589c84e492eb6256ef7a9ae985a, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/5ff55b9f560b44e48bdccde66d20d3b5, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/25adbe40e82e4215a301c73de4368b9f, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/ee4ed16464f04805b19641b5042620ff, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/daa806102d3c481490dcca328e605b44, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/fa26293ed3f54ba8932269a959fd7f4b, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/9442e4d771664dafb9be22d318666831, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/cf93cbe385a94d179af0e33422552b1e, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/066e6b517f3e44cbbb7a82804d7de002, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/1ccaa286287040e9aab6fc56326b943a, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/2936bbce478643c8a70605ac0b915e2a, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/0dcd174fd5394f7eb7fe53a42910ce19, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/a6b7199945d24ebf959dc32ba4d559eb, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/631cae5f1f8e4466898437c28d0a008a, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/d81935ed9ffd46c7a439588a332b12cf, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/95e411e7f0544694b3de909e80e7f789, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/dfbb757ad37546abbadd5f72b1265342, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/17f4fc67a0e24903832774556a38b515, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/434756787fbb49c2919d209e1d3d0bba, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/d6d30be793df4c6080a0736951d3bb0c, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/57b6f1f33e464e4a91e093aab09222e4, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/1a5632fbb8974e6fafcc3a4b22ed1de3, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/1a983a371fe248fb8c289bfd62b4bd56, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/a0ad05fc6a124df8aab5ee7edf4f1c1f, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/e0ef3bf3a1454d7988449aed64df49c7, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/d06ea1455fcf4594acf1567e20d0ef31, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/72ef84f31b4944b5929544771c568edf, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/8c5336167fc74dc3a007f54479d0381d] to archive 2023-04-09 09:54:38,458 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-09 09:54:38,459 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/4f639fd99d8f423c8021f57d15986701 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/4f639fd99d8f423c8021f57d15986701 2023-04-09 09:54:38,460 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/a19da86c8d2b4922a3309a1d5bceb826 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/a19da86c8d2b4922a3309a1d5bceb826 2023-04-09 09:54:38,461 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/15f2f6b0d9f94eaabb88ac1b4c29d88e to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/15f2f6b0d9f94eaabb88ac1b4c29d88e 2023-04-09 09:54:38,462 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/5af34ea9f9e04bf3b298b87dec4719ad to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/5af34ea9f9e04bf3b298b87dec4719ad 2023-04-09 09:54:38,463 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/6c8ea589c84e492eb6256ef7a9ae985a to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/6c8ea589c84e492eb6256ef7a9ae985a 2023-04-09 09:54:38,464 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/5ff55b9f560b44e48bdccde66d20d3b5 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/5ff55b9f560b44e48bdccde66d20d3b5 2023-04-09 09:54:38,465 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/25adbe40e82e4215a301c73de4368b9f to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/25adbe40e82e4215a301c73de4368b9f 2023-04-09 09:54:38,466 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/ee4ed16464f04805b19641b5042620ff to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/ee4ed16464f04805b19641b5042620ff 2023-04-09 09:54:38,467 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/daa806102d3c481490dcca328e605b44 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/daa806102d3c481490dcca328e605b44 2023-04-09 09:54:38,468 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/fa26293ed3f54ba8932269a959fd7f4b to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/fa26293ed3f54ba8932269a959fd7f4b 2023-04-09 09:54:38,469 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/9442e4d771664dafb9be22d318666831 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/9442e4d771664dafb9be22d318666831 2023-04-09 09:54:38,470 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/cf93cbe385a94d179af0e33422552b1e to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/cf93cbe385a94d179af0e33422552b1e 2023-04-09 09:54:38,470 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/066e6b517f3e44cbbb7a82804d7de002 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/066e6b517f3e44cbbb7a82804d7de002 2023-04-09 09:54:38,471 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/1ccaa286287040e9aab6fc56326b943a to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/1ccaa286287040e9aab6fc56326b943a 2023-04-09 09:54:38,472 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/2936bbce478643c8a70605ac0b915e2a to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/2936bbce478643c8a70605ac0b915e2a 2023-04-09 09:54:38,473 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/0dcd174fd5394f7eb7fe53a42910ce19 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/0dcd174fd5394f7eb7fe53a42910ce19 2023-04-09 09:54:38,473 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/a6b7199945d24ebf959dc32ba4d559eb to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/a6b7199945d24ebf959dc32ba4d559eb 2023-04-09 09:54:38,474 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/631cae5f1f8e4466898437c28d0a008a to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/631cae5f1f8e4466898437c28d0a008a 2023-04-09 09:54:38,475 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/d81935ed9ffd46c7a439588a332b12cf to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/d81935ed9ffd46c7a439588a332b12cf 2023-04-09 09:54:38,476 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/95e411e7f0544694b3de909e80e7f789 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/95e411e7f0544694b3de909e80e7f789 2023-04-09 09:54:38,477 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/dfbb757ad37546abbadd5f72b1265342 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/dfbb757ad37546abbadd5f72b1265342 2023-04-09 09:54:38,478 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/17f4fc67a0e24903832774556a38b515 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/17f4fc67a0e24903832774556a38b515 2023-04-09 09:54:38,478 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/434756787fbb49c2919d209e1d3d0bba to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/434756787fbb49c2919d209e1d3d0bba 2023-04-09 09:54:38,479 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/d6d30be793df4c6080a0736951d3bb0c to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/d6d30be793df4c6080a0736951d3bb0c 2023-04-09 09:54:38,480 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/57b6f1f33e464e4a91e093aab09222e4 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/57b6f1f33e464e4a91e093aab09222e4 2023-04-09 09:54:38,481 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/1a5632fbb8974e6fafcc3a4b22ed1de3 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/1a5632fbb8974e6fafcc3a4b22ed1de3 2023-04-09 09:54:38,482 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/1a983a371fe248fb8c289bfd62b4bd56 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/1a983a371fe248fb8c289bfd62b4bd56 2023-04-09 09:54:38,482 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/a0ad05fc6a124df8aab5ee7edf4f1c1f to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/a0ad05fc6a124df8aab5ee7edf4f1c1f 2023-04-09 09:54:38,483 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/e0ef3bf3a1454d7988449aed64df49c7 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/e0ef3bf3a1454d7988449aed64df49c7 2023-04-09 09:54:38,484 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/d06ea1455fcf4594acf1567e20d0ef31 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/d06ea1455fcf4594acf1567e20d0ef31 2023-04-09 09:54:38,485 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/72ef84f31b4944b5929544771c568edf to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/72ef84f31b4944b5929544771c568edf 2023-04-09 09:54:38,486 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/8c5336167fc74dc3a007f54479d0381d to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/8c5336167fc74dc3a007f54479d0381d 2023-04-09 09:54:38,489 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/212195da92fc4eaeadc2aa3ef754b12a, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/dc26da56f5794ff0933d644001475b6c, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/d3c10f3f711f4c979980e64f9a039b3b, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/6b25e4ea2bd94e0c81ce6e6b2e95e215, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/2c38eaa56c724348b2a4338fb35d9e4d, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/f58f3d0e0235434d92830845f1a89175, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/462fc7a5c8ed4c6a9fee53627f78813e, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/d75448fdda2249adb066682e468cfdc6, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/c73c7fdc8e094e2eb4fee221d5f9df93, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/0462cbc291764680b8903cc0b7dd1768, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/cae1801e73734fb895fcd50c971832c9, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/fad46337be894973995dbbafff399d42, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/5effa40e1d1d4c9bb00783c2b6211c39, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/f1ff3b5bae234e8cbffbae27591223a5, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/f524438947784a3aadec83f415494a36, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/e5f9f351093d47979a26310cec35f896, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/a02c593eed0a423990ea56399773ece8, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/6aaaad0d32f3422eabee684a3563ccbb, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/3129816e9c6446468fd588ef4a22f201, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/4eace68c8a0d43b984cf9d16e4257de6, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/0bea334c8be04d178e574792ddf3a786, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/36605c84a13449948e3a9ac065bc563e, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/d30c2449774d483392e3b42c577abc35, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/e3b74271b37f4154ae0bacf5c7924761, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/e5c0b3c60f9a454d878a3c1f4c6097cc, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/f56893d145d348f995e9a9fc16dc9ef5, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/a78005267dfd4b7b9c1cd1bca8d98912, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/ac60ecdb5f394909a93090d48f96627e, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/1e4a3653b11d4297acabcf663054c80f, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/f7816e07b14a40d2a452c96aed6a4e7c, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/4832054364cf41ce875095d57fee191e, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/786961940bb449bfae54ebce35d9b538] to archive 2023-04-09 09:54:38,489 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-09 09:54:38,490 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/212195da92fc4eaeadc2aa3ef754b12a to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/212195da92fc4eaeadc2aa3ef754b12a 2023-04-09 09:54:38,491 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/dc26da56f5794ff0933d644001475b6c to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/dc26da56f5794ff0933d644001475b6c 2023-04-09 09:54:38,492 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/d3c10f3f711f4c979980e64f9a039b3b to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/d3c10f3f711f4c979980e64f9a039b3b 2023-04-09 09:54:38,493 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/6b25e4ea2bd94e0c81ce6e6b2e95e215 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/6b25e4ea2bd94e0c81ce6e6b2e95e215 2023-04-09 09:54:38,494 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/2c38eaa56c724348b2a4338fb35d9e4d to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/2c38eaa56c724348b2a4338fb35d9e4d 2023-04-09 09:54:38,494 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/f58f3d0e0235434d92830845f1a89175 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/f58f3d0e0235434d92830845f1a89175 2023-04-09 09:54:38,495 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/462fc7a5c8ed4c6a9fee53627f78813e to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/462fc7a5c8ed4c6a9fee53627f78813e 2023-04-09 09:54:38,496 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/d75448fdda2249adb066682e468cfdc6 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/d75448fdda2249adb066682e468cfdc6 2023-04-09 09:54:38,497 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/c73c7fdc8e094e2eb4fee221d5f9df93 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/c73c7fdc8e094e2eb4fee221d5f9df93 2023-04-09 09:54:38,497 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/0462cbc291764680b8903cc0b7dd1768 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/0462cbc291764680b8903cc0b7dd1768 2023-04-09 09:54:38,498 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/cae1801e73734fb895fcd50c971832c9 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/cae1801e73734fb895fcd50c971832c9 2023-04-09 09:54:38,499 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/fad46337be894973995dbbafff399d42 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/fad46337be894973995dbbafff399d42 2023-04-09 09:54:38,500 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/5effa40e1d1d4c9bb00783c2b6211c39 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/5effa40e1d1d4c9bb00783c2b6211c39 2023-04-09 09:54:38,501 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/f1ff3b5bae234e8cbffbae27591223a5 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/f1ff3b5bae234e8cbffbae27591223a5 2023-04-09 09:54:38,502 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/f524438947784a3aadec83f415494a36 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/f524438947784a3aadec83f415494a36 2023-04-09 09:54:38,503 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/e5f9f351093d47979a26310cec35f896 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/e5f9f351093d47979a26310cec35f896 2023-04-09 09:54:38,503 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/a02c593eed0a423990ea56399773ece8 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/a02c593eed0a423990ea56399773ece8 2023-04-09 09:54:38,504 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/6aaaad0d32f3422eabee684a3563ccbb to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/6aaaad0d32f3422eabee684a3563ccbb 2023-04-09 09:54:38,505 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/3129816e9c6446468fd588ef4a22f201 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/3129816e9c6446468fd588ef4a22f201 2023-04-09 09:54:38,507 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/4eace68c8a0d43b984cf9d16e4257de6 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/4eace68c8a0d43b984cf9d16e4257de6 2023-04-09 09:54:38,508 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/0bea334c8be04d178e574792ddf3a786 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/0bea334c8be04d178e574792ddf3a786 2023-04-09 09:54:38,509 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/36605c84a13449948e3a9ac065bc563e to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/36605c84a13449948e3a9ac065bc563e 2023-04-09 09:54:38,510 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/d30c2449774d483392e3b42c577abc35 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/d30c2449774d483392e3b42c577abc35 2023-04-09 09:54:38,511 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/e3b74271b37f4154ae0bacf5c7924761 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/e3b74271b37f4154ae0bacf5c7924761 2023-04-09 09:54:38,512 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/e5c0b3c60f9a454d878a3c1f4c6097cc to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/e5c0b3c60f9a454d878a3c1f4c6097cc 2023-04-09 09:54:38,513 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/f56893d145d348f995e9a9fc16dc9ef5 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/f56893d145d348f995e9a9fc16dc9ef5 2023-04-09 09:54:38,514 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/a78005267dfd4b7b9c1cd1bca8d98912 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/a78005267dfd4b7b9c1cd1bca8d98912 2023-04-09 09:54:38,515 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/ac60ecdb5f394909a93090d48f96627e to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/ac60ecdb5f394909a93090d48f96627e 2023-04-09 09:54:38,516 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/1e4a3653b11d4297acabcf663054c80f to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/1e4a3653b11d4297acabcf663054c80f 2023-04-09 09:54:38,516 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/f7816e07b14a40d2a452c96aed6a4e7c to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/f7816e07b14a40d2a452c96aed6a4e7c 2023-04-09 09:54:38,517 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/4832054364cf41ce875095d57fee191e to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/4832054364cf41ce875095d57fee191e 2023-04-09 09:54:38,518 DEBUG [StoreCloser-TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/786961940bb449bfae54ebce35d9b538 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/786961940bb449bfae54ebce35d9b538 2023-04-09 09:54:38,522 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/recovered.edits/653.seqid, newMaxSeqId=653, maxSeqId=4 2023-04-09 09:54:38,524 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40. 2023-04-09 09:54:38,524 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for b86d94f6f111f1359f84891af8d12a40: 2023-04-09 09:54:38,525 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed b86d94f6f111f1359f84891af8d12a40 2023-04-09 09:54:38,526 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=b86d94f6f111f1359f84891af8d12a40, regionState=CLOSED 2023-04-09 09:54:38,526 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681034078526"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681034078526"}]},"ts":"1681034078526"} 2023-04-09 09:54:38,528 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-04-09 09:54:38,528 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; CloseRegionProcedure b86d94f6f111f1359f84891af8d12a40, server=jenkins-hbase4.apache.org,42467,1681034019544 in 1.9630 sec 2023-04-09 09:54:38,529 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-04-09 09:54:38,529 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=b86d94f6f111f1359f84891af8d12a40, UNASSIGN in 1.9680 sec 2023-04-09 09:54:38,529 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681034078529"}]},"ts":"1681034078529"} 2023-04-09 09:54:38,530 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-09 09:54:38,532 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-09 09:54:38,533 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.9810 sec 2023-04-09 09:54:38,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-09 09:54:38,660 INFO [Listener at localhost/44563] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 24 completed 2023-04-09 09:54:38,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-09 09:54:38,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-09 09:54:38,663 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-09 09:54:38,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-09 09:54:38,663 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=27, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-09 09:54:38,668 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40 2023-04-09 09:54:38,670 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A, FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B, FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C, FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/recovered.edits] 2023-04-09 09:54:38,673 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/6f8dfa163b04410d85b9493f2261a20d to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/6f8dfa163b04410d85b9493f2261a20d 2023-04-09 09:54:38,674 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/fa83889a6b5f44ac8b3f2cfa2d0ef17b to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/A/fa83889a6b5f44ac8b3f2cfa2d0ef17b 2023-04-09 09:54:38,676 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/22a462194b514f10b9838bea42c54aaf to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/22a462194b514f10b9838bea42c54aaf 2023-04-09 09:54:38,677 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/4095f8f778554cfc8317e4612e90f841 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/B/4095f8f778554cfc8317e4612e90f841 2023-04-09 09:54:38,679 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/8215b6f2328d40baa754ffb3cdede059 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/8215b6f2328d40baa754ffb3cdede059 2023-04-09 09:54:38,680 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/d68fc25e18a349c0aea09fdd47e2888b to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/C/d68fc25e18a349c0aea09fdd47e2888b 2023-04-09 09:54:38,683 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/recovered.edits/653.seqid to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40/recovered.edits/653.seqid 2023-04-09 09:54:38,684 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/b86d94f6f111f1359f84891af8d12a40 2023-04-09 09:54:38,684 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-09 09:54:38,684 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-09 09:54:38,685 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-09 09:54:38,689 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304090047777d581a43838e3d54bcbce7438a to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304090047777d581a43838e3d54bcbce7438a 2023-04-09 09:54:38,690 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040918844ff1f3a747e1bae9db30d6b215d7 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040918844ff1f3a747e1bae9db30d6b215d7 2023-04-09 09:54:38,692 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040922836ee8020048719ea2cb212d16450e to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040922836ee8020048719ea2cb212d16450e 2023-04-09 09:54:38,693 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409261b98419c1248368e89c746c337efc2 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409261b98419c1248368e89c746c337efc2 2023-04-09 09:54:38,694 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304094c948caa14a1466c94e09bb13635f1ba to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304094c948caa14a1466c94e09bb13635f1ba 2023-04-09 09:54:38,695 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304095090d3e948a44824bf89b17a6745dab4 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304095090d3e948a44824bf89b17a6745dab4 2023-04-09 09:54:38,696 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304096171f39ec73f40568f1950e228f35834 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304096171f39ec73f40568f1950e228f35834 2023-04-09 09:54:38,697 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040964f7b59a1490487481756428ea877142 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040964f7b59a1490487481756428ea877142 2023-04-09 09:54:38,698 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040968230600d4714275ba0d1cf24fed7fad to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040968230600d4714275ba0d1cf24fed7fad 2023-04-09 09:54:38,699 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304096f678fb71bf0414eb80b7dfa5582a104 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304096f678fb71bf0414eb80b7dfa5582a104 2023-04-09 09:54:38,700 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304097f9a45b3867f49d39f3f3d7d8599b902 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304097f9a45b3867f49d39f3f3d7d8599b902 2023-04-09 09:54:38,701 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304097fb5ddb6969f4f0c847e04be9d5a85cd to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304097fb5ddb6969f4f0c847e04be9d5a85cd 2023-04-09 09:54:38,702 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304098750ea7ff0ad4200b9985ae8cb02746f to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304098750ea7ff0ad4200b9985ae8cb02746f 2023-04-09 09:54:38,703 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304098a4eb4ea9c0b43aa851e4d1a47104a91 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304098a4eb4ea9c0b43aa851e4d1a47104a91 2023-04-09 09:54:38,704 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304098b9a3404db8440ce9828194acc60900c to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304098b9a3404db8440ce9828194acc60900c 2023-04-09 09:54:38,705 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304098f7096f5c48d4f3dab5ffdd892c2f2d7 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304098f7096f5c48d4f3dab5ffdd892c2f2d7 2023-04-09 09:54:38,705 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409907e8fca30524771b1424e1526db2d8f to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409907e8fca30524771b1424e1526db2d8f 2023-04-09 09:54:38,707 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409ad9f95631a5248b381fa95650a74c289 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409ad9f95631a5248b381fa95650a74c289 2023-04-09 09:54:38,708 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409b6428e591f6641fdb831f2362703c1c8 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409b6428e591f6641fdb831f2362703c1c8 2023-04-09 09:54:38,709 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409b8f0cff7a1a94e479f3d0454c72c0db9 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409b8f0cff7a1a94e479f3d0454c72c0db9 2023-04-09 09:54:38,710 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409c59175cdae6b4e94aea2e61f041063b8 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409c59175cdae6b4e94aea2e61f041063b8 2023-04-09 09:54:38,711 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409c738f228fb58424c8cd3317dcb65ecfc to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409c738f228fb58424c8cd3317dcb65ecfc 2023-04-09 09:54:38,713 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409c8221449e9264009a11d122a6ee72efe to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409c8221449e9264009a11d122a6ee72efe 2023-04-09 09:54:38,714 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409d3c6457c3d884369b6b32d0b51c29a40 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409d3c6457c3d884369b6b32d0b51c29a40 2023-04-09 09:54:38,715 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409dfde11793f3a4636952443c3860d3e96 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409dfde11793f3a4636952443c3860d3e96 2023-04-09 09:54:38,716 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409f6afd67fd9c84dca97d48373fc0a6a4a to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409f6afd67fd9c84dca97d48373fc0a6a4a 2023-04-09 09:54:38,717 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-09 09:54:38,720 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=27, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-09 09:54:38,722 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-09 09:54:38,724 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-09 09:54:38,725 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=27, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-09 09:54:38,725 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-09 09:54:38,725 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681034078725"}]},"ts":"9223372036854775807"} 2023-04-09 09:54:38,727 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-09 09:54:38,727 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => b86d94f6f111f1359f84891af8d12a40, NAME => 'TestAcidGuarantees,,1681034049808.b86d94f6f111f1359f84891af8d12a40.', STARTKEY => '', ENDKEY => ''}] 2023-04-09 09:54:38,727 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-09 09:54:38,727 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681034078727"}]},"ts":"9223372036854775807"} 2023-04-09 09:54:38,729 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-09 09:54:38,730 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=27, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-09 09:54:38,731 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 70 msec 2023-04-09 09:54:38,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-09 09:54:38,764 INFO [Listener at localhost/44563] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 27 completed 2023-04-09 09:54:38,776 INFO [Listener at localhost/44563] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=234 (was 236), OpenFileDescriptor=559 (was 561), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=433 (was 426) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=1642 (was 1882) 2023-04-09 09:54:38,788 INFO [Listener at localhost/44563] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=234, OpenFileDescriptor=559, MaxFileDescriptor=60000, SystemLoadAverage=433, ProcessCount=169, AvailableMemoryMB=1642 2023-04-09 09:54:38,790 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-09 09:54:38,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-09 09:54:38,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure2.ProcedureExecutor(1029): Stored pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-09 09:54:38,792 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-09 09:54:38,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 28 2023-04-09 09:54:38,793 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-09 09:54:38,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-09 09:54:38,794 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b 2023-04-09 09:54:38,795 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b empty. 2023-04-09 09:54:38,795 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b 2023-04-09 09:54:38,795 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-09 09:54:38,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-09 09:54:39,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-09 09:54:39,203 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-09 09:54:39,204 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 026e485e431b415e1bb021c3d14de88b, NAME => 'TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp 2023-04-09 09:54:39,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-09 09:54:39,610 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-09 09:54:39,610 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 026e485e431b415e1bb021c3d14de88b, disabling compactions & flushes 2023-04-09 09:54:39,610 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:39,610 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:39,610 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. after waiting 0 ms 2023-04-09 09:54:39,610 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:39,610 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:39,610 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 026e485e431b415e1bb021c3d14de88b: 2023-04-09 09:54:39,612 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-09 09:54:39,613 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681034079613"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681034079613"}]},"ts":"1681034079613"} 2023-04-09 09:54:39,614 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-09 09:54:39,615 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-09 09:54:39,615 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681034079615"}]},"ts":"1681034079615"} 2023-04-09 09:54:39,616 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-09 09:54:39,620 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=026e485e431b415e1bb021c3d14de88b, ASSIGN}] 2023-04-09 09:54:39,621 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=026e485e431b415e1bb021c3d14de88b, ASSIGN 2023-04-09 09:54:39,622 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=026e485e431b415e1bb021c3d14de88b, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,42467,1681034019544; forceNewPlan=false, retain=false 2023-04-09 09:54:39,773 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=026e485e431b415e1bb021c3d14de88b, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:39,773 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681034079773"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681034079773"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681034079773"}]},"ts":"1681034079773"} 2023-04-09 09:54:39,775 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; OpenRegionProcedure 026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544}] 2023-04-09 09:54:39,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-09 09:54:39,930 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:39,930 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 026e485e431b415e1bb021c3d14de88b, NAME => 'TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.', STARTKEY => '', ENDKEY => ''} 2023-04-09 09:54:39,930 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 026e485e431b415e1bb021c3d14de88b 2023-04-09 09:54:39,930 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-09 09:54:39,930 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 026e485e431b415e1bb021c3d14de88b 2023-04-09 09:54:39,930 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 026e485e431b415e1bb021c3d14de88b 2023-04-09 09:54:39,931 INFO [StoreOpener-026e485e431b415e1bb021c3d14de88b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 026e485e431b415e1bb021c3d14de88b 2023-04-09 09:54:39,932 DEBUG [StoreOpener-026e485e431b415e1bb021c3d14de88b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A 2023-04-09 09:54:39,933 DEBUG [StoreOpener-026e485e431b415e1bb021c3d14de88b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A 2023-04-09 09:54:39,933 INFO [StoreOpener-026e485e431b415e1bb021c3d14de88b-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-09 09:54:39,933 INFO [StoreOpener-026e485e431b415e1bb021c3d14de88b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 026e485e431b415e1bb021c3d14de88b columnFamilyName A 2023-04-09 09:54:39,933 INFO [StoreOpener-026e485e431b415e1bb021c3d14de88b-1] regionserver.HStore(310): Store=026e485e431b415e1bb021c3d14de88b/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-09 09:54:39,934 INFO [StoreOpener-026e485e431b415e1bb021c3d14de88b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 026e485e431b415e1bb021c3d14de88b 2023-04-09 09:54:39,935 DEBUG [StoreOpener-026e485e431b415e1bb021c3d14de88b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B 2023-04-09 09:54:39,935 DEBUG [StoreOpener-026e485e431b415e1bb021c3d14de88b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B 2023-04-09 09:54:39,935 INFO [StoreOpener-026e485e431b415e1bb021c3d14de88b-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-09 09:54:39,935 INFO [StoreOpener-026e485e431b415e1bb021c3d14de88b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 026e485e431b415e1bb021c3d14de88b columnFamilyName B 2023-04-09 09:54:39,935 INFO [StoreOpener-026e485e431b415e1bb021c3d14de88b-1] regionserver.HStore(310): Store=026e485e431b415e1bb021c3d14de88b/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-09 09:54:39,935 INFO [StoreOpener-026e485e431b415e1bb021c3d14de88b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 026e485e431b415e1bb021c3d14de88b 2023-04-09 09:54:39,936 DEBUG [StoreOpener-026e485e431b415e1bb021c3d14de88b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C 2023-04-09 09:54:39,936 DEBUG [StoreOpener-026e485e431b415e1bb021c3d14de88b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C 2023-04-09 09:54:39,937 INFO [StoreOpener-026e485e431b415e1bb021c3d14de88b-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-09 09:54:39,937 INFO [StoreOpener-026e485e431b415e1bb021c3d14de88b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 026e485e431b415e1bb021c3d14de88b columnFamilyName C 2023-04-09 09:54:39,937 INFO [StoreOpener-026e485e431b415e1bb021c3d14de88b-1] regionserver.HStore(310): Store=026e485e431b415e1bb021c3d14de88b/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-09 09:54:39,937 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:39,938 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b 2023-04-09 09:54:39,938 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b 2023-04-09 09:54:39,939 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-09 09:54:39,940 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 026e485e431b415e1bb021c3d14de88b 2023-04-09 09:54:39,942 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-09 09:54:39,942 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 026e485e431b415e1bb021c3d14de88b; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65274621, jitterRate=-0.027332350611686707}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-09 09:54:39,942 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 026e485e431b415e1bb021c3d14de88b: 2023-04-09 09:54:39,943 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b., pid=30, masterSystemTime=1681034079926 2023-04-09 09:54:39,944 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:39,944 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:39,945 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=026e485e431b415e1bb021c3d14de88b, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:39,945 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681034079945"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681034079945"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681034079945"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681034079945"}]},"ts":"1681034079945"} 2023-04-09 09:54:39,947 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-04-09 09:54:39,947 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; OpenRegionProcedure 026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 in 171 msec 2023-04-09 09:54:39,948 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-04-09 09:54:39,948 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=026e485e431b415e1bb021c3d14de88b, ASSIGN in 327 msec 2023-04-09 09:54:39,949 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-09 09:54:39,949 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681034079949"}]},"ts":"1681034079949"} 2023-04-09 09:54:39,949 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-09 09:54:39,951 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-09 09:54:39,952 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=28, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1610 sec 2023-04-09 09:54:40,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-09 09:54:40,898 INFO [Listener at localhost/44563] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 28 completed 2023-04-09 09:54:40,898 DEBUG [Listener at localhost/44563] zookeeper.ReadOnlyZKClient(139): Connect 0x22b4eae5 to 127.0.0.1:57816 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-09 09:54:40,903 DEBUG [Listener at localhost/44563] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@12e82edd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-09 09:54:40,904 DEBUG [Listener at localhost/44563] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-09 09:54:40,906 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56072, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-09 09:54:40,907 DEBUG [Listener at localhost/44563] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-09 09:54:40,909 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36996, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-09 09:54:40,910 DEBUG [Listener at localhost/44563] zookeeper.ReadOnlyZKClient(139): Connect 0x262f4570 to 127.0.0.1:57816 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-09 09:54:40,915 DEBUG [Listener at localhost/44563] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@23c97092, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-09 09:54:40,915 DEBUG [Listener at localhost/44563] zookeeper.ReadOnlyZKClient(139): Connect 0x00ca898c to 127.0.0.1:57816 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-09 09:54:40,919 DEBUG [Listener at localhost/44563] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5c124820, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-09 09:54:40,919 DEBUG [Listener at localhost/44563] zookeeper.ReadOnlyZKClient(139): Connect 0x6585a70a to 127.0.0.1:57816 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-09 09:54:40,923 DEBUG [Listener at localhost/44563] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3157d4df, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-09 09:54:40,923 DEBUG [Listener at localhost/44563] zookeeper.ReadOnlyZKClient(139): Connect 0x7480f9da to 127.0.0.1:57816 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-09 09:54:40,926 DEBUG [Listener at localhost/44563] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3cf5bdb7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-09 09:54:40,927 DEBUG [Listener at localhost/44563] zookeeper.ReadOnlyZKClient(139): Connect 0x1a24d617 to 127.0.0.1:57816 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-09 09:54:40,933 DEBUG [Listener at localhost/44563] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6919b2a0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-09 09:54:40,933 DEBUG [Listener at localhost/44563] zookeeper.ReadOnlyZKClient(139): Connect 0x2ce5573e to 127.0.0.1:57816 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-09 09:54:40,938 DEBUG [Listener at localhost/44563] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@8cd7cb0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-09 09:54:40,938 DEBUG [Listener at localhost/44563] zookeeper.ReadOnlyZKClient(139): Connect 0x087b3095 to 127.0.0.1:57816 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-09 09:54:40,942 DEBUG [Listener at localhost/44563] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@ea1ca03, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-09 09:54:40,942 DEBUG [Listener at localhost/44563] zookeeper.ReadOnlyZKClient(139): Connect 0x12115052 to 127.0.0.1:57816 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-09 09:54:40,946 DEBUG [Listener at localhost/44563] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@433f251d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-09 09:54:40,946 DEBUG [Listener at localhost/44563] zookeeper.ReadOnlyZKClient(139): Connect 0x7e65578e to 127.0.0.1:57816 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-09 09:54:40,951 DEBUG [Listener at localhost/44563] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@66ee0bd0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-09 09:54:40,951 DEBUG [Listener at localhost/44563] zookeeper.ReadOnlyZKClient(139): Connect 0x6fcbaba8 to 127.0.0.1:57816 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-09 09:54:40,957 DEBUG [Listener at localhost/44563] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3b4658f8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-09 09:54:40,962 DEBUG [hconnection-0x3f402a7e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-09 09:54:40,963 DEBUG [hconnection-0x4e098777-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-09 09:54:40,963 DEBUG [hconnection-0x3e9c6a08-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-09 09:54:40,963 DEBUG [hconnection-0x2efa8a09-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-09 09:54:40,963 DEBUG [hconnection-0x27f3e05d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-09 09:54:40,964 DEBUG [hconnection-0x88bd240-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-09 09:54:40,964 DEBUG [hconnection-0x6311252f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-09 09:54:40,965 DEBUG [hconnection-0x3fb0ead7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-09 09:54:40,965 DEBUG [hconnection-0x6713ef5e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-09 09:54:40,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:40,967 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56098, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-09 09:54:40,968 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56078, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-09 09:54:40,968 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56106, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-09 09:54:40,965 DEBUG [hconnection-0x29778257-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-09 09:54:40,968 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56124, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-09 09:54:40,973 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56080, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-09 09:54:40,973 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56110, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-09 09:54:40,974 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56154, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-09 09:54:40,978 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56126, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-09 09:54:40,978 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56138, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-09 09:54:40,979 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56168, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-09 09:54:40,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:40,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:40,980 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:40,980 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:40,980 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:40,980 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:40,981 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:40,981 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:40,983 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:40,983 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:40,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:40,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:40,983 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:40,983 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:40,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:40,983 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:40,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:40,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:40,984 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:40,984 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:40,985 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:40,986 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:40,986 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:40,986 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:40,986 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:40,986 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:40,986 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:40,986 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. started... 2023-04-09 09:54:40,997 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 026e485e431b415e1bb021c3d14de88b 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-09 09:54:40,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 026e485e431b415e1bb021c3d14de88b 2023-04-09 09:54:40,998 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=A 2023-04-09 09:54:40,998 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. as already flushing 2023-04-09 09:54:40,998 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:40,998 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=B 2023-04-09 09:54:40,998 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:40,998 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=C 2023-04-09 09:54:40,998 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:41,018 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:41,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 5 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56168 deadline: 1681034141014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,018 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:41,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56106 deadline: 1681034141014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,019 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:41,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56126 deadline: 1681034141015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,019 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681034081011, max timestamp=1681034081016Num uniques 859; 2023-04-09 09:54:41,019 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-09 09:54:41,019 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681034081011, max timestamp=1681034081016Num uniques 859; 2023-04-09 09:54:41,019 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-09 09:54:41,019 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681034081011, max timestamp=1681034081016Num uniques 859; 2023-04-09 09:54:41,019 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-09 09:54:41,023 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/A/f502705a26b545028f6d111a3e52184f 2023-04-09 09:54:41,046 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:41,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56098 deadline: 1681034141045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,048 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:41,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56138 deadline: 1681034141047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,122 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:41,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56106 deadline: 1681034141121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,122 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:41,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56126 deadline: 1681034141121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,122 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:41,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56168 deadline: 1681034141121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,149 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:41,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56098 deadline: 1681034141148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,150 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:41,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56138 deadline: 1681034141149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,284 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-09 09:54:41,324 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:41,324 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:41,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56106 deadline: 1681034141323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56126 deadline: 1681034141323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,325 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:41,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56168 deadline: 1681034141323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,351 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:41,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56098 deadline: 1681034141350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,353 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:41,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56138 deadline: 1681034141352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,446 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/B/a256341ad74447948ce54982a841198c 2023-04-09 09:54:41,475 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/C/4814e1f19e2e4ed48dde22c63190a534 2023-04-09 09:54:41,491 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/A/f502705a26b545028f6d111a3e52184f as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/f502705a26b545028f6d111a3e52184f 2023-04-09 09:54:41,496 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/f502705a26b545028f6d111a3e52184f, entries=150, sequenceid=12, filesize=11.6 K 2023-04-09 09:54:41,498 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/B/a256341ad74447948ce54982a841198c as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/a256341ad74447948ce54982a841198c 2023-04-09 09:54:41,503 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/a256341ad74447948ce54982a841198c, entries=150, sequenceid=12, filesize=11.6 K 2023-04-09 09:54:41,504 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/C/4814e1f19e2e4ed48dde22c63190a534 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/4814e1f19e2e4ed48dde22c63190a534 2023-04-09 09:54:41,514 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/4814e1f19e2e4ed48dde22c63190a534, entries=150, sequenceid=12, filesize=11.6 K 2023-04-09 09:54:41,515 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 026e485e431b415e1bb021c3d14de88b in 518ms, sequenceid=12, compaction requested=false 2023-04-09 09:54:41,515 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 026e485e431b415e1bb021c3d14de88b: 2023-04-09 09:54:41,515 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:41,515 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:41,515 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:41,515 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,515 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:41,515 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:41,518 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,518 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:41,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:41,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:41,518 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:41,518 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:41,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:41,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:41,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:41,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:41,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:41,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@650dffef[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:41,521 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:41,521 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:41,522 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:41,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:41,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:41,522 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:41,522 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:41,522 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:41,522 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,522 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:41,524 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:41,524 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:41,524 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,524 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:41,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:41,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:41,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:41,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:41,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:41,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:41,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:41,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:41,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:41,527 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:41,527 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:41,527 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:41,527 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:41,527 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:41,529 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:41,529 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:41,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:41,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:41,529 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:41,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:41,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:41,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:41,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:41,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:41,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:41,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:41,530 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:41,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:41,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:41,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,532 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:41,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:41,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,536 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,536 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:41,536 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:41,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:41,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:41,536 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:41,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:41,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:41,536 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:41,536 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,536 DEBUG [Thread-1715] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:41,536 DEBUG [Thread-1715] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:41,536 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:41,536 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:41,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:41,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:41,536 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:41,537 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:41,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 026e485e431b415e1bb021c3d14de88b 2023-04-09 09:54:41,628 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 026e485e431b415e1bb021c3d14de88b 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-09 09:54:41,629 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=A 2023-04-09 09:54:41,630 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-09 09:54:41,630 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=B 2023-04-09 09:54:41,630 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-09 09:54:41,630 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=C 2023-04-09 09:54:41,630 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-09 09:54:41,636 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:41,636 DEBUG [Thread-1715] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:41,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56106 deadline: 1681034141634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:41,638 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:41,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56126 deadline: 1681034141636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,639 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:41,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56168 deadline: 1681034141637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:41,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:41,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:41,641 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:41,641 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:41,642 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:41,642 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:41,642 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:41,642 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:41,643 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:41,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:41,643 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:41,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:41,644 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,644 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:41,644 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:41,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:41,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:41,645 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:41,645 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:41,645 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/A/57bb5d34e0b246c78e148df37acf6353 2023-04-09 09:54:41,645 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:41,645 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:41,645 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:41,645 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:41,646 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:41,646 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:41,646 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:41,646 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. started... 2023-04-09 09:54:41,646 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. as already flushing 2023-04-09 09:54:41,646 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:41,646 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:41,646 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:41,646 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,646 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:41,646 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:41,648 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,648 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:41,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:41,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:41,648 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:41,649 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:41,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:41,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:41,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:41,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:41,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:41,650 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:41,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32fabb22[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:41,650 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:41,651 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:41,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:41,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:41,652 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:41,652 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:41,652 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:41,653 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:41,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56098 deadline: 1681034141652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,655 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:41,655 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,655 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:41,655 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:41,655 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:41,655 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:41,655 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:41,655 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:41,655 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:41,655 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:41,655 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:41,657 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:41,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56138 deadline: 1681034141656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,660 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:41,660 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:41,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:41,660 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:41,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:41,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:41,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:41,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:41,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:41,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:41,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:41,661 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:41,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:41,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:41,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:41,662 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:41,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:41,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,664 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/B/a2ff42ee6fe94b72a1ab2dce64120332 2023-04-09 09:54:41,665 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:41,665 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:41,665 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:41,665 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:41,665 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:41,665 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,665 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:41,665 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:41,665 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:41,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:41,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:41,666 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:41,665 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:41,665 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:41,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:41,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:41,666 DEBUG [Thread-1715] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:41,666 DEBUG [Thread-1715] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:41,738 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:41,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56106 deadline: 1681034141738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,740 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:41,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56126 deadline: 1681034141739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,740 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:41,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56168 deadline: 1681034141740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,766 DEBUG [Thread-1715] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:41,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:41,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:41,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:41,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:41,770 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:41,770 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:41,770 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:41,770 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:41,770 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:41,770 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:41,771 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,771 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:41,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:41,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:41,772 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,772 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:41,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:41,772 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:41,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:41,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:41,773 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:41,773 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:41,773 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:41,773 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:41,773 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:41,773 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:41,773 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:41,773 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:41,773 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:41,773 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. started... 2023-04-09 09:54:41,774 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. as already flushing 2023-04-09 09:54:41,774 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:41,774 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:41,774 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:41,774 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,774 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:41,774 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:41,776 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,776 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:41,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:41,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:41,776 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:41,776 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:41,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:41,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:41,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:41,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:41,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:41,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7079463[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:41,778 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:41,778 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:41,779 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:41,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:41,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:41,779 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:41,779 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:41,779 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,779 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:41,779 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:41,780 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,780 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:41,780 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:41,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:41,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:41,780 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:41,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:41,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:41,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:41,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:41,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:41,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:41,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:41,783 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:41,783 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:41,783 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:41,783 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:41,783 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:41,784 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:41,784 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:41,784 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:41,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:41,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:41,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:41,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:41,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:41,785 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:41,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:41,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:41,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:41,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:41,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:41,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:41,786 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:41,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:41,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,788 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,788 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:41,788 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:41,788 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:41,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:41,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:41,788 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:41,788 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:41,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:41,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:41,789 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:41,788 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,789 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:41,789 DEBUG [Thread-1715] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:41,789 DEBUG [Thread-1715] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:41,789 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:41,789 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:41,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:41,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:41,889 DEBUG [Thread-1715] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:41,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:41,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:41,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:41,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:41,892 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:41,892 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:41,892 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:41,893 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:41,893 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:41,893 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:41,894 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:41,895 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:41,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:41,895 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,895 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:41,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:41,895 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:41,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:41,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:41,896 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:41,896 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:41,896 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:41,896 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:41,896 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:41,896 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:41,896 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:41,896 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:41,896 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:41,896 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. started... 2023-04-09 09:54:41,896 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. as already flushing 2023-04-09 09:54:41,897 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:41,897 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:41,897 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:41,897 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,897 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:41,897 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:41,898 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,898 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:41,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:41,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:41,899 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:41,899 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:41,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:41,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:41,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:41,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:41,900 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:41,900 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ae880ad[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:41,900 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:41,900 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:41,902 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,902 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:41,902 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:41,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:41,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:41,902 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:41,902 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:41,902 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:41,905 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:41,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:41,905 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:41,905 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:41,905 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:41,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:41,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:41,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:41,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:41,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:41,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:41,907 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:41,907 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:41,907 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:41,907 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:41,907 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:41,907 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:41,909 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:41,909 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:41,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:41,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:41,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:41,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:41,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:41,909 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:41,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:41,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:41,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:41,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:41,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:41,910 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:41,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:41,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,911 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:41,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:41,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,913 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,913 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:41,914 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:41,914 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,914 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:41,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:41,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:41,914 DEBUG [Thread-1715] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:41,914 DEBUG [Thread-1715] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:41,914 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:41,914 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:41,914 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:41,915 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:41,915 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:41,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:41,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:41,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:41,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:41,941 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:41,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56106 deadline: 1681034141940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,943 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:41,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56126 deadline: 1681034141941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:41,943 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:41,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56168 deadline: 1681034141941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:42,014 DEBUG [Thread-1715] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:42,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:42,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:42,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:42,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:42,018 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:42,018 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:42,018 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:42,019 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:42,019 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:42,019 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:42,020 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:42,020 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:42,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:42,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:42,021 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:42,021 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:42,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:42,021 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:42,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:42,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:42,021 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:42,022 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:42,022 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:42,022 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:42,022 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:42,022 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:42,022 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:42,022 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:42,022 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:42,022 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. started... 2023-04-09 09:54:42,022 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. as already flushing 2023-04-09 09:54:42,022 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:42,022 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:42,022 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:42,022 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:42,022 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:42,023 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:42,025 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:42,025 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:42,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:42,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:42,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:42,025 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:42,025 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:42,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:42,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:42,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:42,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:42,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:42,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:42,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6090f898[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:42,026 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:42,027 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:42,028 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:42,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:42,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:42,028 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:42,028 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:42,028 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:42,028 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:42,028 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:42,029 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:42,029 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:42,029 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:42,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:42,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:42,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:42,029 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:42,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:42,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:42,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:42,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:42,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:42,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:42,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:42,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:42,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:42,031 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:42,031 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:42,031 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:42,031 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:42,031 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:42,033 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:42,033 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:42,033 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:42,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:42,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:42,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:42,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:42,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:42,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:42,034 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:42,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:42,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:42,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:42,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:42,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:42,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:42,035 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:42,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:42,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:42,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:42,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:42,043 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:42,043 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:42,044 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:42,043 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:42,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:42,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:42,044 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:42,044 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:42,044 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:42,044 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:42,044 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:42,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:42,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:42,044 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:42,044 DEBUG [Thread-1715] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:42,044 DEBUG [Thread-1715] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:42,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:42,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:42,075 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/C/246a21a132c942a29f36740d4561825e 2023-04-09 09:54:42,080 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/A/57bb5d34e0b246c78e148df37acf6353 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/57bb5d34e0b246c78e148df37acf6353 2023-04-09 09:54:42,083 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/57bb5d34e0b246c78e148df37acf6353, entries=200, sequenceid=50, filesize=13.9 K 2023-04-09 09:54:42,084 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/B/a2ff42ee6fe94b72a1ab2dce64120332 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/a2ff42ee6fe94b72a1ab2dce64120332 2023-04-09 09:54:42,087 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/a2ff42ee6fe94b72a1ab2dce64120332, entries=150, sequenceid=50, filesize=11.6 K 2023-04-09 09:54:42,088 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/C/246a21a132c942a29f36740d4561825e as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/246a21a132c942a29f36740d4561825e 2023-04-09 09:54:42,092 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/246a21a132c942a29f36740d4561825e, entries=150, sequenceid=50, filesize=11.6 K 2023-04-09 09:54:42,092 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~420.80 KB/430896, currentSize=33.54 KB/34350 for 026e485e431b415e1bb021c3d14de88b in 464ms, sequenceid=50, compaction requested=false 2023-04-09 09:54:42,092 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 026e485e431b415e1bb021c3d14de88b: 2023-04-09 09:54:42,144 DEBUG [Thread-1715] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:42,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:42,149 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:42,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:42,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:42,151 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:42,151 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:42,151 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:42,151 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:42,151 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:42,151 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:42,153 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:42,153 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:42,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:42,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:42,153 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:42,153 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:42,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:42,153 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:42,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:42,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:42,154 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:42,154 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:42,154 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:42,154 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:42,154 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:42,154 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:42,155 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:42,155 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:42,155 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:42,155 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. started... 2023-04-09 09:54:42,155 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 026e485e431b415e1bb021c3d14de88b 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-09 09:54:42,155 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=A 2023-04-09 09:54:42,155 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:42,155 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=B 2023-04-09 09:54:42,156 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:42,156 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=C 2023-04-09 09:54:42,156 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:42,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 026e485e431b415e1bb021c3d14de88b 2023-04-09 09:54:42,163 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. as already flushing 2023-04-09 09:54:42,191 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034082183, max timestamp=1681034082189Num uniques 859; 2023-04-09 09:54:42,191 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:54:42,191 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034082183, max timestamp=1681034082189Num uniques 859; 2023-04-09 09:54:42,191 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034082183, max timestamp=1681034082189Num uniques 859; 2023-04-09 09:54:42,191 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:54:42,192 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:54:42,217 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681034082204, max timestamp=1681034082215Num uniques 1018; 2023-04-09 09:54:42,217 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-09 09:54:42,218 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681034082204, max timestamp=1681034082215Num uniques 1018; 2023-04-09 09:54:42,218 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-09 09:54:42,218 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681034082204, max timestamp=1681034082215Num uniques 1018; 2023-04-09 09:54:42,218 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-09 09:54:42,249 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:42,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56098 deadline: 1681034142246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:42,249 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:42,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56168 deadline: 1681034142246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:42,250 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681034082226, max timestamp=1681034082247Num uniques 1077; 2023-04-09 09:54:42,250 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-09 09:54:42,250 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8092, totalHeapSize=18648, min timestamp=1681034082226, max timestamp=1681034082247Num uniques 1077; 2023-04-09 09:54:42,250 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8092, totalHeapSize=18648, min timestamp=1681034082226, max timestamp=1681034082247Num uniques 1077; 2023-04-09 09:54:42,250 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49312, new segment data size=8092, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-09 09:54:42,250 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49312, new segment data size=8092, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-09 09:54:42,275 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681034082261, max timestamp=1681034082273Num uniques 1036; 2023-04-09 09:54:42,276 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-09 09:54:42,278 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681034082261, max timestamp=1681034082273Num uniques 1036; 2023-04-09 09:54:42,278 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-09 09:54:42,280 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681034082261, max timestamp=1681034082273Num uniques 1036; 2023-04-09 09:54:42,280 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=6870, suffix heap size=135712,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=150 2023-04-09 09:54:42,296 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:42,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56126 deadline: 1681034142293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:42,296 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1681034082288, max timestamp=1681034082293Num uniques 1045; 2023-04-09 09:54:42,296 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-09 09:54:42,296 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681034082288, max timestamp=1681034082293Num uniques 1045; 2023-04-09 09:54:42,296 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681034082288, max timestamp=1681034082293Num uniques 1009; 2023-04-09 09:54:42,296 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-09 09:54:42,296 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=8930, suffix heap size=132904,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=195 2023-04-09 09:54:42,359 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1681034082346, max timestamp=1681034082357Num uniques 1104; 2023-04-09 09:54:42,359 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1681034082346, max timestamp=1681034082357Num uniques 1054; 2023-04-09 09:54:42,359 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=11630, suffix heap size=142784,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=254 2023-04-09 09:54:42,359 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=11630, suffix heap size=137584,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=254 2023-04-09 09:54:42,360 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1681034082346, max timestamp=1681034082357Num uniques 1054; 2023-04-09 09:54:42,360 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=11630, suffix heap size=137584,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=254 2023-04-09 09:54:42,382 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:42,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56098 deadline: 1681034142379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:42,382 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:42,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56138 deadline: 1681034142380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:42,398 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:42,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56106 deadline: 1681034142381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:42,400 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:42,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56126 deadline: 1681034142397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:42,484 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:42,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56098 deadline: 1681034142483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:42,499 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:42,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56138 deadline: 1681034142497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:42,500 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:42,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56106 deadline: 1681034142499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:42,574 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/A/1ef21a481a224b2a992ee1e3e0a1d751 2023-04-09 09:54:42,585 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/B/d99c438e56864157a835fb143ad7a8aa 2023-04-09 09:54:42,603 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:42,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56126 deadline: 1681034142603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:42,693 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:42,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56098 deadline: 1681034142689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:42,702 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:42,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56138 deadline: 1681034142701, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:42,704 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:42,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56106 deadline: 1681034142701, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:42,753 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:42,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56168 deadline: 1681034142751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:42,908 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:42,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56126 deadline: 1681034142906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:42,998 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:42,998 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/C/20c157f0ff8b4cb496b16a85c08b55b2 2023-04-09 09:54:42,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56098 deadline: 1681034142996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:43,002 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/A/1ef21a481a224b2a992ee1e3e0a1d751 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/1ef21a481a224b2a992ee1e3e0a1d751 2023-04-09 09:54:43,006 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/1ef21a481a224b2a992ee1e3e0a1d751, entries=150, sequenceid=58, filesize=11.6 K 2023-04-09 09:54:43,007 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:43,007 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/B/d99c438e56864157a835fb143ad7a8aa as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/d99c438e56864157a835fb143ad7a8aa 2023-04-09 09:54:43,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56106 deadline: 1681034143005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:43,007 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:43,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56138 deadline: 1681034143006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:43,011 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/d99c438e56864157a835fb143ad7a8aa, entries=150, sequenceid=58, filesize=11.6 K 2023-04-09 09:54:43,011 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/C/20c157f0ff8b4cb496b16a85c08b55b2 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/20c157f0ff8b4cb496b16a85c08b55b2 2023-04-09 09:54:43,015 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/20c157f0ff8b4cb496b16a85c08b55b2, entries=150, sequenceid=58, filesize=11.6 K 2023-04-09 09:54:43,016 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=147.60 KB/151140 for 026e485e431b415e1bb021c3d14de88b in 861ms, sequenceid=58, compaction requested=true 2023-04-09 09:54:43,016 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 026e485e431b415e1bb021c3d14de88b: 2023-04-09 09:54:43,016 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:43,016 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:43,016 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:43,016 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:43,016 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:43,016 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:43,018 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:43,018 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:43,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:43,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:43,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:43,019 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:43,019 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:43,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:43,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:43,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:43,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:43,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:43,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:43,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d46c03[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:43,020 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:43,020 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:43,022 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:43,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:43,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:43,022 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:43,022 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:43,022 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:43,022 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:43,022 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:43,023 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:43,023 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:43,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:43,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:43,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:43,023 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:43,023 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:43,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:43,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:43,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:43,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:43,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:43,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:43,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:43,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:43,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:43,026 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:43,026 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:43,026 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:43,026 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:43,026 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:43,028 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:43,028 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:43,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:43,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:43,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:43,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:43,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:43,029 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:43,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:43,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:43,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:43,029 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:43,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:43,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:43,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:43,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:43,031 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:43,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:43,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:43,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:43,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:43,034 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:43,034 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:43,034 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:43,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:43,035 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:43,035 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:43,034 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:43,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:43,035 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:43,035 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:43,035 DEBUG [Thread-1715] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:43,035 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:43,035 DEBUG [Thread-1715] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:43,035 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:43,035 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:43,035 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:43,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:43,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:43,135 DEBUG [Thread-1715] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:43,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:43,137 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:43,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:43,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:43,138 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:43,138 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:43,138 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:43,138 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:43,139 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:43,139 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:43,140 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:43,140 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:43,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:43,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:43,141 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:43,141 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:43,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:43,141 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:43,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:43,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:43,141 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:43,141 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:43,142 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:43,142 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:43,142 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:43,142 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:43,142 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:43,142 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:43,142 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:43,142 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. started... 2023-04-09 09:54:43,142 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 026e485e431b415e1bb021c3d14de88b 3/3 column families, dataSize=147.60 KB heapSize=415.92 KB 2023-04-09 09:54:43,142 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=A 2023-04-09 09:54:43,142 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:54:43,142 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=B 2023-04-09 09:54:43,142 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:54:43,142 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=C 2023-04-09 09:54:43,143 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:54:43,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 026e485e431b415e1bb021c3d14de88b 2023-04-09 09:54:43,414 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. as already flushing 2023-04-09 09:54:43,427 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:43,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56126 deadline: 1681034143425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:43,504 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:43,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56098 deadline: 1681034143502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:43,511 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:43,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56138 deadline: 1681034143508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:43,511 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:43,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56106 deadline: 1681034143509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:43,531 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:43,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56126 deadline: 1681034143528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:43,553 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=181 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/A/608b0420f8ef4396958cfcf2ba6b8b67 2023-04-09 09:54:43,735 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:43,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56126 deadline: 1681034143733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:43,760 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:43,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56168 deadline: 1681034143757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:43,964 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=181 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/B/120fd20790654e6698fecd9d6728c6ea 2023-04-09 09:54:43,988 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=181 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/C/b5f0f964faf24fb2955bc8056dc955d4 2023-04-09 09:54:43,993 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/A/608b0420f8ef4396958cfcf2ba6b8b67 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/608b0420f8ef4396958cfcf2ba6b8b67 2023-04-09 09:54:43,997 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/608b0420f8ef4396958cfcf2ba6b8b67, entries=150, sequenceid=181, filesize=11.8 K 2023-04-09 09:54:43,998 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/B/120fd20790654e6698fecd9d6728c6ea as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/120fd20790654e6698fecd9d6728c6ea 2023-04-09 09:54:44,002 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/120fd20790654e6698fecd9d6728c6ea, entries=150, sequenceid=181, filesize=11.8 K 2023-04-09 09:54:44,002 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/C/b5f0f964faf24fb2955bc8056dc955d4 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/b5f0f964faf24fb2955bc8056dc955d4 2023-04-09 09:54:44,006 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/b5f0f964faf24fb2955bc8056dc955d4, entries=150, sequenceid=181, filesize=11.8 K 2023-04-09 09:54:44,006 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~415.88 KB/425856, currentSize=33.54 KB/34350 for 026e485e431b415e1bb021c3d14de88b in 864ms, sequenceid=181, compaction requested=true 2023-04-09 09:54:44,007 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 026e485e431b415e1bb021c3d14de88b: 2023-04-09 09:54:44,007 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:44,007 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:44,007 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:44,007 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,007 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:44,007 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:44,008 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:44,008 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:44,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:44,009 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:44,009 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:44,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:44,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:44,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:44,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:44,010 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:44,010 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c4b5416[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:44,010 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:44,010 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:44,012 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:44,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:44,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:44,012 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:44,012 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:44,012 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:44,013 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,013 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:44,014 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,014 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:44,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:44,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:44,014 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:44,014 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:44,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:44,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:44,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:44,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:44,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:44,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:44,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:44,016 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:44,016 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:44,016 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:44,016 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:44,016 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:44,018 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:44,018 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:44,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:44,018 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:44,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:44,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:44,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:44,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:44,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:44,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:44,018 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:44,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:44,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:44,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:44,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:44,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,020 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:44,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:44,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,022 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,022 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:44,022 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:44,022 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:44,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:44,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:44,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:44,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:44,022 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:44,022 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:44,022 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:44,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:44,023 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:44,023 DEBUG [Thread-1715] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:44,023 DEBUG [Thread-1715] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:44,023 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:44,023 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:44,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 026e485e431b415e1bb021c3d14de88b 2023-04-09 09:54:44,041 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 026e485e431b415e1bb021c3d14de88b 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-09 09:54:44,041 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=A 2023-04-09 09:54:44,041 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:44,041 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=B 2023-04-09 09:54:44,041 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:44,041 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=C 2023-04-09 09:54:44,041 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:44,053 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-09 09:54:44,099 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034084079, max timestamp=1681034084097Num uniques 859; 2023-04-09 09:54:44,099 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034084079, max timestamp=1681034084097Num uniques 859; 2023-04-09 09:54:44,099 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:54:44,099 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:54:44,100 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034084079, max timestamp=1681034084097Num uniques 859; 2023-04-09 09:54:44,100 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:54:44,123 DEBUG [Thread-1715] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:44,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:44,125 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:44,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:44,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:44,126 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:44,126 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:44,126 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:44,126 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:44,127 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:44,127 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:44,129 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:44,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:44,129 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:44,129 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,129 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:44,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:44,129 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:44,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:44,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:44,130 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:44,130 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:44,130 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:44,130 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:44,130 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:44,130 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:44,130 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:44,130 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:44,130 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:44,130 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. started... 2023-04-09 09:54:44,131 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. as already flushing 2023-04-09 09:54:44,131 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:44,131 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:44,131 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:44,131 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,131 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:44,131 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:44,132 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,132 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:44,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:44,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:44,133 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:44,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:44,133 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:44,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:44,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:44,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:44,134 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:44,134 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@128846da[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:44,134 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:44,134 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:44,135 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:44,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:44,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:44,135 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,135 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:44,135 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:44,135 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:44,135 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:44,137 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:44,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:44,137 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:44,137 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:44,137 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:44,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:44,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:44,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:44,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:44,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:44,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:44,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:44,140 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:44,140 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:44,140 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:44,140 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:44,140 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:44,142 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:44,142 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:44,142 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:44,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:44,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:44,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:44,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:44,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:44,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:44,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:44,143 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:44,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:44,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:44,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:44,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:44,144 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:44,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:44,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,148 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,148 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:44,148 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:44,148 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:44,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:44,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:44,148 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:44,149 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:44,148 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:44,149 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:44,149 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:44,149 DEBUG [Thread-1715] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:44,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:44,149 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:44,149 DEBUG [Thread-1715] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:44,149 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:44,149 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:44,149 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:44,158 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:44,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56126 deadline: 1681034144155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,249 DEBUG [Thread-1715] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:44,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:44,252 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:44,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:44,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:44,261 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:44,261 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:44,261 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:44,261 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:44,261 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:44,261 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:44,261 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:44,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56126 deadline: 1681034144259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,262 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:44,262 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:44,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:44,263 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,263 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:44,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:44,263 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:44,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:44,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:44,264 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:44,264 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:44,264 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:44,264 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:44,264 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:44,264 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:44,265 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:44,265 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:44,265 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:44,265 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. started... 2023-04-09 09:54:44,265 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. as already flushing 2023-04-09 09:54:44,265 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:44,265 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:44,265 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:44,265 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,265 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:44,265 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:44,266 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,267 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:44,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:44,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:44,267 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:44,267 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:44,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:44,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:44,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:44,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:44,268 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:44,268 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75bb382b[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:44,268 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:44,268 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:44,269 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:44,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:44,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:44,270 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:44,270 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:44,270 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:44,270 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,270 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:44,272 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,272 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:44,272 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:44,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:44,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:44,272 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:44,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:44,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:44,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:44,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:44,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:44,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:44,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:44,274 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:44,274 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:44,274 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:44,274 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:44,274 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:44,276 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:44,276 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:44,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:44,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:44,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:44,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:44,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:44,276 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:44,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:44,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:44,276 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:44,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:44,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:44,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:44,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:44,277 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:44,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:44,279 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,279 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:44,279 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:44,279 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:44,279 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:44,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:44,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:44,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:44,279 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:44,279 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:44,280 DEBUG [Thread-1715] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:44,280 DEBUG [Thread-1715] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:44,280 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:44,280 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:44,280 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:44,280 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:44,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:44,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:44,380 DEBUG [Thread-1715] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:44,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:44,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:44,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:44,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:44,383 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:44,383 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:44,384 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:44,384 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:44,384 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:44,384 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:44,385 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:44,385 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:44,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:44,386 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,386 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:44,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:44,386 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:44,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:44,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:44,387 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:44,387 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:44,387 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:44,387 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:44,387 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:44,387 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:44,387 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:44,387 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:44,387 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:44,387 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. started... 2023-04-09 09:54:44,388 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. as already flushing 2023-04-09 09:54:44,388 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:44,388 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:44,388 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:44,388 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,388 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:44,388 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:44,390 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,390 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:44,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:44,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:44,390 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:44,391 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:44,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:44,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:44,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:44,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:44,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:44,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@512be6eb[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:44,392 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:44,392 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:44,393 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:44,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:44,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:44,393 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:44,393 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:44,393 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:44,393 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,393 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:44,394 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,394 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:44,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:44,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:44,394 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:44,395 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:44,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:44,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:44,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:44,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:44,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:44,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:44,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:44,397 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:44,397 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:44,397 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:44,397 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:44,397 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:44,399 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:44,399 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:44,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:44,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:44,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:44,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:44,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:44,400 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:44,400 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:44,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:44,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:44,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:44,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:44,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:44,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:44,402 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:44,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:44,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,405 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:44,405 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:44,405 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:44,405 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:44,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:44,405 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:44,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:44,405 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:44,406 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:44,406 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,406 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:44,406 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:44,406 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:44,406 DEBUG [Thread-1715] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:44,406 DEBUG [Thread-1715] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:44,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:44,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:44,448 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=191 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/A/2622df9731e54b299d0370686a2ba6c0 2023-04-09 09:54:44,464 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:44,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56126 deadline: 1681034144463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,506 DEBUG [Thread-1715] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:44,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:44,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:44,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:44,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:44,509 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:44,509 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:44,509 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:44,509 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:44,510 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:44,510 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:44,511 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,511 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:44,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:44,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:44,511 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,511 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:44,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:44,512 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:44,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:44,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:44,512 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:44,512 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:44,512 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:44,512 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:44,512 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:44,512 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:44,512 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:44,513 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:44,513 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:44,513 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. started... 2023-04-09 09:54:44,513 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. as already flushing 2023-04-09 09:54:44,513 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:44,513 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:44,513 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:44,513 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,513 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:44,513 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:44,514 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,514 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:44,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:44,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:44,515 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:44,515 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:44,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:44,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:44,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:44,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:44,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:44,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3726f33c[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:44,516 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:44,516 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:44,517 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:44,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:44,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:44,517 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,517 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:44,517 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:44,517 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:44,517 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:44,517 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:44,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56106 deadline: 1681034144513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,518 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:44,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56098 deadline: 1681034144514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,520 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,520 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:44,520 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:44,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:44,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:44,520 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:44,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:44,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:44,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:44,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:44,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:44,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:44,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:44,522 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:44,522 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:44,522 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:44,522 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:44,522 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:44,523 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:44,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56138 deadline: 1681034144520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,523 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:44,523 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:44,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:44,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:44,524 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:44,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:44,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:44,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:44,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:44,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:44,524 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:44,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:44,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:44,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:44,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:44,525 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:44,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:44,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,527 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,527 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:44,527 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:44,527 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,527 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:44,528 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:44,528 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:44,528 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:44,528 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:44,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:44,528 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:44,528 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:44,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:44,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:44,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:44,537 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:44,537 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:44,537 DEBUG [Thread-1715] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:44,537 DEBUG [Thread-1715] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:44,637 DEBUG [Thread-1715] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:44,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:44,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:44,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:44,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:44,641 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:44,641 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:44,641 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:44,641 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:44,641 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:44,641 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:44,645 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,645 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:44,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:44,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:44,645 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,646 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:44,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:44,646 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:44,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:44,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:44,646 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:44,647 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:44,647 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:44,647 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:44,647 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:44,647 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:44,647 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:44,647 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:44,647 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:44,647 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. started... 2023-04-09 09:54:44,647 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. as already flushing 2023-04-09 09:54:44,647 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:44,647 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:44,647 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:44,647 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,647 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:44,647 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:44,649 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,649 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:44,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:44,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:44,649 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:44,649 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:44,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:44,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:44,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:44,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:44,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:44,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6748dd94[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:44,650 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:44,651 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:44,651 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:44,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:44,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:44,652 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:44,652 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:44,652 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,652 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:44,652 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:44,653 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:44,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:44,653 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:44,653 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:44,654 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:44,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:44,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:44,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:44,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:44,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:44,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:44,655 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:44,655 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:44,656 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:44,656 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:44,656 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:44,656 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:44,658 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:44,658 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:44,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:44,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:44,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:44,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:44,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:44,659 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:44,659 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:44,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:44,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:44,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:44,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:44,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:44,660 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:44,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:44,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:44,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,662 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:44,663 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:44,663 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,663 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:44,663 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:44,663 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,663 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:44,663 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:44,663 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:44,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:44,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:44,663 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:44,663 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:44,663 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:44,663 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:44,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:44,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:44,664 DEBUG [Thread-1715] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:44,664 DEBUG [Thread-1715] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:44,764 DEBUG [Thread-1715] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:44,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:44,766 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:44,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:44,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:44,768 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:44,768 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:44,768 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:44,768 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:44,769 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:44,769 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:44,770 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:44,770 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:44,770 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:44,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:44,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56126 deadline: 1681034144768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:44,771 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:44,771 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,771 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:44,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:44,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:44,772 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:44,772 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:44,772 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:44,772 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:44,772 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:44,772 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:44,776 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:44,776 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:44,776 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:44,776 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. started... 2023-04-09 09:54:44,776 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. as already flushing 2023-04-09 09:54:44,776 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:44,777 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:44,777 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:44,777 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,777 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:44,777 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:44,779 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,779 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:44,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:44,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:44,779 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:44,779 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:44,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:44,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:44,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:44,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:44,781 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:44,781 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b1edb36[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:44,781 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:44,782 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:44,784 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:44,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:44,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:44,784 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:44,784 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:44,784 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:44,784 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,784 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:44,786 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:44,786 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:44,786 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:44,789 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:44,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:44,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:44,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:44,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:44,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:44,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:44,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:44,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:44,799 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:44,799 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:44,799 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:44,799 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:44,799 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:44,802 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:44,802 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:44,802 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:44,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:44,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:44,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:44,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:44,803 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:44,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:44,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:44,804 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:44,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:44,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,806 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,806 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:44,807 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:44,806 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:44,807 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:44,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:44,807 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,807 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:44,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:44,807 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:44,807 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:44,807 DEBUG [Thread-1715] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:44,807 DEBUG [Thread-1715] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:44,807 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:44,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:44,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:44,808 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:44,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:44,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:44,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:44,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:44,859 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=191 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/B/acf69788d8ef43eea271b334d0105ff6 2023-04-09 09:54:44,882 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=191 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/C/0b2d5aa00056494b90135fb04ed7d5e0 2023-04-09 09:54:44,887 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/A/2622df9731e54b299d0370686a2ba6c0 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/2622df9731e54b299d0370686a2ba6c0 2023-04-09 09:54:44,906 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/2622df9731e54b299d0370686a2ba6c0, entries=200, sequenceid=191, filesize=14.1 K 2023-04-09 09:54:44,907 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/B/acf69788d8ef43eea271b334d0105ff6 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/acf69788d8ef43eea271b334d0105ff6 2023-04-09 09:54:44,907 DEBUG [Thread-1715] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:44,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:44,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:44,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:44,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:44,919 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:44,920 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:44,920 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:44,920 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:44,920 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:44,920 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:44,923 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:44,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:44,923 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:44,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:44,924 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:44,924 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:44,924 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:44,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:44,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:44,925 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:44,926 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/acf69788d8ef43eea271b334d0105ff6, entries=150, sequenceid=191, filesize=11.8 K 2023-04-09 09:54:44,927 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:44,927 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:44,927 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/C/0b2d5aa00056494b90135fb04ed7d5e0 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/0b2d5aa00056494b90135fb04ed7d5e0 2023-04-09 09:54:44,927 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:44,927 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:44,927 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:44,942 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/0b2d5aa00056494b90135fb04ed7d5e0, entries=150, sequenceid=191, filesize=11.8 K 2023-04-09 09:54:44,942 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 026e485e431b415e1bb021c3d14de88b in 901ms, sequenceid=191, compaction requested=true 2023-04-09 09:54:44,943 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 026e485e431b415e1bb021c3d14de88b: 2023-04-09 09:54:44,943 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:54:44,943 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:54:44,943 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-09 09:54:44,943 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-09 09:54:44,943 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:54:44,944 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:44,944 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:44,944 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:44,944 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. started... 2023-04-09 09:54:44,944 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 026e485e431b415e1bb021c3d14de88b 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-09 09:54:44,944 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=A 2023-04-09 09:54:44,945 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:54:44,945 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=B 2023-04-09 09:54:44,945 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59730 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-09 09:54:44,945 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 64460 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-09 09:54:44,945 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): 026e485e431b415e1bb021c3d14de88b/B is initiating minor compaction (all files) 2023-04-09 09:54:44,945 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1912): 026e485e431b415e1bb021c3d14de88b/A is initiating minor compaction (all files) 2023-04-09 09:54:44,945 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:54:44,945 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 026e485e431b415e1bb021c3d14de88b/A in TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:44,945 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=C 2023-04-09 09:54:44,945 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 026e485e431b415e1bb021c3d14de88b/B in TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:44,945 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/f502705a26b545028f6d111a3e52184f, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/57bb5d34e0b246c78e148df37acf6353, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/1ef21a481a224b2a992ee1e3e0a1d751, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/608b0420f8ef4396958cfcf2ba6b8b67, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/2622df9731e54b299d0370686a2ba6c0] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp, totalSize=62.9 K 2023-04-09 09:54:44,945 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:54:44,945 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/a256341ad74447948ce54982a841198c, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/a2ff42ee6fe94b72a1ab2dce64120332, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/d99c438e56864157a835fb143ad7a8aa, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/120fd20790654e6698fecd9d6728c6ea, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/acf69788d8ef43eea271b334d0105ff6] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp, totalSize=58.3 K 2023-04-09 09:54:44,946 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting f502705a26b545028f6d111a3e52184f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1681034080982 2023-04-09 09:54:44,946 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting a256341ad74447948ce54982a841198c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1681034080982 2023-04-09 09:54:44,946 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 57bb5d34e0b246c78e148df37acf6353, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1681034081043 2023-04-09 09:54:44,947 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting a2ff42ee6fe94b72a1ab2dce64120332, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1681034081043 2023-04-09 09:54:44,947 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 1ef21a481a224b2a992ee1e3e0a1d751, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1681034081632 2023-04-09 09:54:44,948 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting d99c438e56864157a835fb143ad7a8aa, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1681034081632 2023-04-09 09:54:44,948 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 608b0420f8ef4396958cfcf2ba6b8b67, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1681034082374 2023-04-09 09:54:44,948 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 120fd20790654e6698fecd9d6728c6ea, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1681034082374 2023-04-09 09:54:44,949 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 2622df9731e54b299d0370686a2ba6c0, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=191, earliestPutTs=1681034083421 2023-04-09 09:54:44,949 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting acf69788d8ef43eea271b334d0105ff6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=191, earliestPutTs=1681034083424 2023-04-09 09:54:44,974 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): 026e485e431b415e1bb021c3d14de88b#B#compaction#232 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:54:44,985 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] throttle.PressureAwareThroughputController(145): 026e485e431b415e1bb021c3d14de88b#A#compaction#233 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:54:45,005 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/B/d05195cb2cb94afeb94d7ca6494119fb as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/d05195cb2cb94afeb94d7ca6494119fb 2023-04-09 09:54:45,006 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/A/c2f492d8b26140288a2859a72d3f9fd8 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/c2f492d8b26140288a2859a72d3f9fd8 2023-04-09 09:54:45,017 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 026e485e431b415e1bb021c3d14de88b/B of 026e485e431b415e1bb021c3d14de88b into d05195cb2cb94afeb94d7ca6494119fb(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:54:45,018 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 026e485e431b415e1bb021c3d14de88b: 2023-04-09 09:54:45,018 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b., storeName=026e485e431b415e1bb021c3d14de88b/B, priority=11, startTime=1681034084943; duration=0sec 2023-04-09 09:54:45,018 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:54:45,018 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-09 09:54:45,018 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 026e485e431b415e1bb021c3d14de88b/A of 026e485e431b415e1bb021c3d14de88b into c2f492d8b26140288a2859a72d3f9fd8(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:54:45,019 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 026e485e431b415e1bb021c3d14de88b: 2023-04-09 09:54:45,019 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b., storeName=026e485e431b415e1bb021c3d14de88b/A, priority=11, startTime=1681034084943; duration=0sec 2023-04-09 09:54:45,019 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:54:45,020 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59730 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-09 09:54:45,020 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): 026e485e431b415e1bb021c3d14de88b/C is initiating minor compaction (all files) 2023-04-09 09:54:45,020 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 026e485e431b415e1bb021c3d14de88b/C in TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:45,020 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/4814e1f19e2e4ed48dde22c63190a534, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/246a21a132c942a29f36740d4561825e, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/20c157f0ff8b4cb496b16a85c08b55b2, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/b5f0f964faf24fb2955bc8056dc955d4, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/0b2d5aa00056494b90135fb04ed7d5e0] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp, totalSize=58.3 K 2023-04-09 09:54:45,032 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 4814e1f19e2e4ed48dde22c63190a534, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1681034080982 2023-04-09 09:54:45,032 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 246a21a132c942a29f36740d4561825e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1681034081043 2023-04-09 09:54:45,033 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 20c157f0ff8b4cb496b16a85c08b55b2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1681034081632 2023-04-09 09:54:45,033 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting b5f0f964faf24fb2955bc8056dc955d4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1681034082374 2023-04-09 09:54:45,035 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 0b2d5aa00056494b90135fb04ed7d5e0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=191, earliestPutTs=1681034083424 2023-04-09 09:54:45,053 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): 026e485e431b415e1bb021c3d14de88b#C#compaction#234 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:54:45,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 026e485e431b415e1bb021c3d14de88b 2023-04-09 09:54:45,276 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. as already flushing 2023-04-09 09:54:45,296 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:45,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56126 deadline: 1681034145294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:45,387 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=228 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/A/f57fb7e9664445deb2c2a5a9bc067f27 2023-04-09 09:54:45,400 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:45,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56126 deadline: 1681034145398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:45,405 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=228 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/B/196396ca09244474bddbb4bbe6a5b6e4 2023-04-09 09:54:45,470 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/C/62f0b22ba5e94888b26c3ad48c981f28 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/62f0b22ba5e94888b26c3ad48c981f28 2023-04-09 09:54:45,475 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 026e485e431b415e1bb021c3d14de88b/C of 026e485e431b415e1bb021c3d14de88b into 62f0b22ba5e94888b26c3ad48c981f28(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:54:45,475 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 026e485e431b415e1bb021c3d14de88b: 2023-04-09 09:54:45,475 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b., storeName=026e485e431b415e1bb021c3d14de88b/C, priority=11, startTime=1681034084943; duration=0sec 2023-04-09 09:54:45,475 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:54:45,603 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:45,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56126 deadline: 1681034145601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:45,774 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:45,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56168 deadline: 1681034145773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:45,776 DEBUG [Thread-1709] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4139 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b., hostname=jenkins-hbase4.apache.org,42467,1681034019544, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-09 09:54:45,816 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=228 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/C/d06e77d285e946f19aa3a65434c45337 2023-04-09 09:54:45,821 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/A/f57fb7e9664445deb2c2a5a9bc067f27 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/f57fb7e9664445deb2c2a5a9bc067f27 2023-04-09 09:54:45,826 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/f57fb7e9664445deb2c2a5a9bc067f27, entries=150, sequenceid=228, filesize=11.8 K 2023-04-09 09:54:45,826 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/B/196396ca09244474bddbb4bbe6a5b6e4 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/196396ca09244474bddbb4bbe6a5b6e4 2023-04-09 09:54:45,830 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/196396ca09244474bddbb4bbe6a5b6e4, entries=150, sequenceid=228, filesize=11.8 K 2023-04-09 09:54:45,831 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/C/d06e77d285e946f19aa3a65434c45337 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/d06e77d285e946f19aa3a65434c45337 2023-04-09 09:54:45,835 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/d06e77d285e946f19aa3a65434c45337, entries=150, sequenceid=228, filesize=11.8 K 2023-04-09 09:54:45,836 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 026e485e431b415e1bb021c3d14de88b in 892ms, sequenceid=228, compaction requested=false 2023-04-09 09:54:45,836 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 026e485e431b415e1bb021c3d14de88b: 2023-04-09 09:54:45,836 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:45,836 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:45,836 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:45,836 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:45,840 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:45,840 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:45,841 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:45,841 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:45,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:45,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:45,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:45,842 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:45,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:45,842 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:45,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:45,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:45,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:45,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:45,843 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:45,843 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@458c6000[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:45,843 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:45,843 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:45,846 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:45,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:45,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:45,846 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:45,846 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:45,846 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:45,846 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:45,847 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:45,848 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:45,848 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:45,848 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:45,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:45,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:45,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:45,848 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:45,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:45,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:45,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:45,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:45,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:45,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:45,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:45,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:45,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:45,851 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:45,851 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:45,851 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:45,851 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:45,851 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:45,853 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:45,853 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:45,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:45,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:45,853 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:45,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:45,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:45,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:45,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:45,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:45,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:45,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:45,855 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:45,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:45,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:45,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:45,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:45,856 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:45,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:45,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:45,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:45,859 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:45,859 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:45,859 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:45,859 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:45,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:45,859 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:45,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:45,859 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:45,859 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:45,859 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:45,859 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:45,859 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:45,859 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:45,859 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:45,859 DEBUG [Thread-1715] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:45,859 DEBUG [Thread-1715] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:45,860 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:45,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:45,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:45,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 026e485e431b415e1bb021c3d14de88b 2023-04-09 09:54:45,908 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 026e485e431b415e1bb021c3d14de88b 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-09 09:54:45,908 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=A 2023-04-09 09:54:45,909 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:45,909 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=B 2023-04-09 09:54:45,909 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:45,909 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=C 2023-04-09 09:54:45,909 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:45,915 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=242 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/A/a893115462b04945b4d93807770c6c3a 2023-04-09 09:54:45,957 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034085931, max timestamp=1681034085955Num uniques 859; 2023-04-09 09:54:45,957 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034085931, max timestamp=1681034085955Num uniques 859; 2023-04-09 09:54:45,957 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:54:45,957 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034085931, max timestamp=1681034085955Num uniques 859; 2023-04-09 09:54:45,957 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:54:45,957 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:54:45,960 DEBUG [Thread-1715] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:45,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:45,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:45,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:45,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:45,963 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:45,963 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:45,963 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:45,963 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:45,963 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:45,963 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:45,970 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:45,970 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:45,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:45,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:45,971 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:45,971 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:45,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:45,971 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:45,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:45,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:45,971 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:45,972 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:45,972 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:45,972 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:45,972 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:45,972 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:45,972 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:45,972 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:45,972 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:45,972 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. started... 2023-04-09 09:54:45,972 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. as already flushing 2023-04-09 09:54:45,972 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:45,972 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:45,972 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:45,972 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:45,972 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:45,972 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:45,974 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:45,974 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:45,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:45,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:45,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:45,974 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:45,974 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:45,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:45,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:45,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:45,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:45,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:45,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:45,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4533c49e[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:45,976 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:45,976 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:45,980 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:45,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:45,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:45,980 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:45,980 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:45,980 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:45,980 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:45,980 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:45,982 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:45,982 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:45,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:45,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:45,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:45,982 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:45,982 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:45,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:45,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:45,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:45,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:45,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:45,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:45,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:45,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:45,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:45,985 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:45,985 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:45,985 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:45,985 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:45,985 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:45,987 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:45,987 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:45,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:45,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:45,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:45,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:45,987 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:45,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:45,987 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:45,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:45,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:45,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:45,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:45,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:45,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:45,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:45,988 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:45,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:45,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:45,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:45,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:45,991 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:45,991 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:45,991 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:45,991 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:45,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:45,991 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:45,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:45,991 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:45,991 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:45,991 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:45,991 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:45,991 DEBUG [Thread-1715] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:45,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:45,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:45,991 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:45,991 DEBUG [Thread-1715] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:45,991 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:45,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:45,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:46,004 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:46,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56126 deadline: 1681034146002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,091 DEBUG [Thread-1715] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:46,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:46,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:46,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:46,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:46,095 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:46,095 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:46,095 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:46,095 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:46,095 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:46,095 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:46,096 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:46,097 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:46,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:46,097 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,097 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:46,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:46,097 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:46,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:46,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:46,098 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:46,098 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:46,098 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:46,098 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:46,098 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:46,098 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:46,098 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:46,098 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:46,098 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:46,099 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. started... 2023-04-09 09:54:46,099 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. as already flushing 2023-04-09 09:54:46,099 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:46,099 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:46,099 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:46,099 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,099 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:46,099 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:46,102 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,102 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:46,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:46,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:46,102 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:46,102 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:46,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:46,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:46,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:46,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:46,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:46,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@643615d1[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:46,103 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:46,103 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:46,104 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:46,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:46,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:46,105 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:46,105 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,105 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:46,105 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:46,105 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:46,106 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:46,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:46,106 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:46,106 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:46,106 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:46,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:46,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:46,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:46,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:46,108 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:46,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:46,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56126 deadline: 1681034146105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:46,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:46,109 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:46,109 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:46,109 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:46,109 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:46,109 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:46,110 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:46,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:46,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:46,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:46,111 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:46,111 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:46,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:46,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:46,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:46,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:46,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:46,111 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:46,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:46,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:46,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:46,113 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:46,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:46,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,115 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,115 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:46,115 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:46,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:46,115 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:46,115 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:46,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:46,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:46,116 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:46,115 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:46,116 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,116 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:46,116 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:46,116 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:46,116 DEBUG [Thread-1715] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:46,116 DEBUG [Thread-1715] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:46,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:46,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:46,216 DEBUG [Thread-1715] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:46,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:46,219 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:46,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:46,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:46,220 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:46,220 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:46,220 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:46,220 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:46,220 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:46,220 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:46,222 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,222 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:46,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:46,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:46,222 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,222 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:46,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:46,222 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:46,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:46,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:46,223 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:46,223 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:46,223 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:46,223 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:46,223 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:46,223 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:46,223 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:46,223 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:46,223 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:46,224 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. started... 2023-04-09 09:54:46,224 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. as already flushing 2023-04-09 09:54:46,224 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:46,224 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:46,224 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:46,224 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,224 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:46,224 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:46,225 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:46,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:46,225 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:46,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:46,226 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:46,226 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:46,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:46,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:46,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:46,227 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:46,227 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59cca50a[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:46,227 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:46,227 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:46,228 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:46,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:46,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:46,228 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:46,228 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:46,228 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:46,229 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,229 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:46,231 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:46,231 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,231 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:46,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:46,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:46,231 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:46,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:46,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:46,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:46,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:46,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:46,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:46,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:46,233 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:46,234 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:46,234 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:46,234 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:46,234 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:46,235 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:46,235 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:46,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:46,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:46,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:46,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:46,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:46,235 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:46,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:46,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:46,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:46,236 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:46,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:46,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:46,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:46,237 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:46,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:46,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,239 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,239 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:46,239 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:46,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:46,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:46,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:46,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:46,239 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:46,239 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:46,240 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,240 DEBUG [Thread-1715] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:46,240 DEBUG [Thread-1715] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:46,239 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:46,240 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:46,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:46,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:46,240 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:46,240 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:46,310 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:46,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56126 deadline: 1681034146309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,325 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=242 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/B/5eefd7952e6f494bbddf84b13487ebb2 2023-04-09 09:54:46,336 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=242 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/C/476073b5e2f9448e9b9730d4ea71d047 2023-04-09 09:54:46,340 DEBUG [Thread-1715] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:46,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:46,341 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/A/a893115462b04945b4d93807770c6c3a as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/a893115462b04945b4d93807770c6c3a 2023-04-09 09:54:46,343 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:46,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:46,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:46,345 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:46,345 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:46,345 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:46,345 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:46,346 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:46,346 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:46,346 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/a893115462b04945b4d93807770c6c3a, entries=150, sequenceid=242, filesize=11.8 K 2023-04-09 09:54:46,347 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:46,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:46,347 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/B/5eefd7952e6f494bbddf84b13487ebb2 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/5eefd7952e6f494bbddf84b13487ebb2 2023-04-09 09:54:46,347 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:46,348 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,348 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:46,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:46,348 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:46,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:46,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:46,349 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:46,349 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:46,349 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:46,349 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:46,349 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:46,349 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:46,349 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:46,349 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:46,349 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:46,350 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. started... 2023-04-09 09:54:46,350 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. as already flushing 2023-04-09 09:54:46,350 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:46,350 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:46,350 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:46,350 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,350 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:46,350 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:46,351 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/5eefd7952e6f494bbddf84b13487ebb2, entries=150, sequenceid=242, filesize=11.8 K 2023-04-09 09:54:46,352 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/C/476073b5e2f9448e9b9730d4ea71d047 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/476073b5e2f9448e9b9730d4ea71d047 2023-04-09 09:54:46,354 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,354 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:46,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:46,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:46,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:46,354 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:46,354 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:46,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:46,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:46,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:46,356 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:46,357 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f665253[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:46,357 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:46,357 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:46,357 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/476073b5e2f9448e9b9730d4ea71d047, entries=150, sequenceid=242, filesize=11.8 K 2023-04-09 09:54:46,358 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 026e485e431b415e1bb021c3d14de88b in 450ms, sequenceid=242, compaction requested=true 2023-04-09 09:54:46,358 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 026e485e431b415e1bb021c3d14de88b: 2023-04-09 09:54:46,358 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:54:46,358 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-09 09:54:46,358 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-09 09:54:46,358 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:54:46,358 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:46,358 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:54:46,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:46,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:46,358 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:46,358 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:46,359 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:46,359 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,359 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:46,360 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36280 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-09 09:54:46,360 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36280 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-09 09:54:46,360 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): 026e485e431b415e1bb021c3d14de88b/B is initiating minor compaction (all files) 2023-04-09 09:54:46,360 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1912): 026e485e431b415e1bb021c3d14de88b/A is initiating minor compaction (all files) 2023-04-09 09:54:46,360 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 026e485e431b415e1bb021c3d14de88b/B in TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:46,360 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 026e485e431b415e1bb021c3d14de88b/A in TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:46,360 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/d05195cb2cb94afeb94d7ca6494119fb, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/196396ca09244474bddbb4bbe6a5b6e4, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/5eefd7952e6f494bbddf84b13487ebb2] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp, totalSize=35.4 K 2023-04-09 09:54:46,360 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/c2f492d8b26140288a2859a72d3f9fd8, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/f57fb7e9664445deb2c2a5a9bc067f27, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/a893115462b04945b4d93807770c6c3a] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp, totalSize=35.4 K 2023-04-09 09:54:46,360 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting d05195cb2cb94afeb94d7ca6494119fb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=191, earliestPutTs=1681034083424 2023-04-09 09:54:46,361 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting c2f492d8b26140288a2859a72d3f9fd8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=191, earliestPutTs=1681034083424 2023-04-09 09:54:46,361 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 196396ca09244474bddbb4bbe6a5b6e4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=228, earliestPutTs=1681034084120 2023-04-09 09:54:46,361 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting f57fb7e9664445deb2c2a5a9bc067f27, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=228, earliestPutTs=1681034084120 2023-04-09 09:54:46,361 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 5eefd7952e6f494bbddf84b13487ebb2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=242, earliestPutTs=1681034085284 2023-04-09 09:54:46,361 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting a893115462b04945b4d93807770c6c3a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=242, earliestPutTs=1681034085284 2023-04-09 09:54:46,362 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:46,362 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,362 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:46,362 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:46,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:46,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:46,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:46,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:46,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:46,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:46,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:46,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:46,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:46,366 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:46,366 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:46,366 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:46,366 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:46,366 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:46,368 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] throttle.PressureAwareThroughputController(145): 026e485e431b415e1bb021c3d14de88b#A#compaction#241 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:54:46,368 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): 026e485e431b415e1bb021c3d14de88b#B#compaction#240 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:54:46,370 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:46,370 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:46,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:46,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:46,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:46,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:46,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:46,370 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:46,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:46,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:46,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:46,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:46,371 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:46,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:46,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:46,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,372 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:46,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:46,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,374 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,374 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:46,374 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:46,374 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:46,374 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:46,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:46,374 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:46,375 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:46,374 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:46,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:46,375 DEBUG [Thread-1715] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:46,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:46,375 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:46,375 DEBUG [Thread-1715] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:46,375 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:46,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:46,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:46,475 DEBUG [Thread-1715] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:46,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:46,477 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:46,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:46,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:46,478 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:46,478 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:46,478 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:46,478 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:46,479 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:46,479 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:46,480 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:46,480 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:46,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:46,481 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,481 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:46,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:46,481 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:46,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:46,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:46,481 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:46,482 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:46,482 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:46,482 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:46,482 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:46,482 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:46,482 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:46,482 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:46,482 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:46,482 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. started... 2023-04-09 09:54:46,482 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 026e485e431b415e1bb021c3d14de88b 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-09 09:54:46,483 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=A 2023-04-09 09:54:46,483 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:54:46,483 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=B 2023-04-09 09:54:46,483 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:54:46,483 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=C 2023-04-09 09:54:46,483 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:54:46,490 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=278 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/A/b034770880d242788150337c0cc951c4 2023-04-09 09:54:46,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 026e485e431b415e1bb021c3d14de88b 2023-04-09 09:54:46,534 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. as already flushing 2023-04-09 09:54:46,544 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:46,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56138 deadline: 1681034146539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,544 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:46,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56106 deadline: 1681034146542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,547 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:46,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56098 deadline: 1681034146544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,614 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:46,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56126 deadline: 1681034146613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,646 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:46,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56138 deadline: 1681034146645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,647 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:46,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56106 deadline: 1681034146645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,650 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:46,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56098 deadline: 1681034146648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,789 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/A/c3866e6c1489427c9bc6d2f16462952a as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/c3866e6c1489427c9bc6d2f16462952a 2023-04-09 09:54:46,792 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/B/6db091ddcac54a84a2b8fa9921b85016 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/6db091ddcac54a84a2b8fa9921b85016 2023-04-09 09:54:46,794 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 026e485e431b415e1bb021c3d14de88b/A of 026e485e431b415e1bb021c3d14de88b into c3866e6c1489427c9bc6d2f16462952a(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:54:46,794 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 026e485e431b415e1bb021c3d14de88b: 2023-04-09 09:54:46,794 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b., storeName=026e485e431b415e1bb021c3d14de88b/A, priority=13, startTime=1681034086358; duration=0sec 2023-04-09 09:54:46,794 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:54:46,794 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-09 09:54:46,795 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36280 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-09 09:54:46,795 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1912): 026e485e431b415e1bb021c3d14de88b/C is initiating minor compaction (all files) 2023-04-09 09:54:46,795 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 026e485e431b415e1bb021c3d14de88b/C in TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:46,796 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/62f0b22ba5e94888b26c3ad48c981f28, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/d06e77d285e946f19aa3a65434c45337, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/476073b5e2f9448e9b9730d4ea71d047] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp, totalSize=35.4 K 2023-04-09 09:54:46,796 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 62f0b22ba5e94888b26c3ad48c981f28, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=191, earliestPutTs=1681034083424 2023-04-09 09:54:46,797 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting d06e77d285e946f19aa3a65434c45337, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=228, earliestPutTs=1681034084120 2023-04-09 09:54:46,797 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 476073b5e2f9448e9b9730d4ea71d047, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=242, earliestPutTs=1681034085284 2023-04-09 09:54:46,797 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 026e485e431b415e1bb021c3d14de88b/B of 026e485e431b415e1bb021c3d14de88b into 6db091ddcac54a84a2b8fa9921b85016(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:54:46,797 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 026e485e431b415e1bb021c3d14de88b: 2023-04-09 09:54:46,797 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b., storeName=026e485e431b415e1bb021c3d14de88b/B, priority=13, startTime=1681034086358; duration=0sec 2023-04-09 09:54:46,797 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:54:46,802 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] throttle.PressureAwareThroughputController(145): 026e485e431b415e1bb021c3d14de88b#C#compaction#244 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:54:46,848 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:46,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56138 deadline: 1681034146848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,849 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:46,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56106 deadline: 1681034146849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,852 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:46,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56098 deadline: 1681034146851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,902 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=278 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/B/713ce4b41b3348b78d88385a5fce2c4c 2023-04-09 09:54:46,922 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=278 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/C/f38a2531c1144f3fae0c8a9ad08d16ed 2023-04-09 09:54:46,926 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/A/b034770880d242788150337c0cc951c4 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/b034770880d242788150337c0cc951c4 2023-04-09 09:54:46,931 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/b034770880d242788150337c0cc951c4, entries=150, sequenceid=278, filesize=11.9 K 2023-04-09 09:54:46,932 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/B/713ce4b41b3348b78d88385a5fce2c4c as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/713ce4b41b3348b78d88385a5fce2c4c 2023-04-09 09:54:46,944 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/713ce4b41b3348b78d88385a5fce2c4c, entries=150, sequenceid=278, filesize=11.9 K 2023-04-09 09:54:46,945 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/C/f38a2531c1144f3fae0c8a9ad08d16ed as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/f38a2531c1144f3fae0c8a9ad08d16ed 2023-04-09 09:54:46,950 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/f38a2531c1144f3fae0c8a9ad08d16ed, entries=150, sequenceid=278, filesize=11.9 K 2023-04-09 09:54:46,950 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 026e485e431b415e1bb021c3d14de88b in 468ms, sequenceid=278, compaction requested=false 2023-04-09 09:54:46,950 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 026e485e431b415e1bb021c3d14de88b: 2023-04-09 09:54:46,951 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:46,951 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:46,951 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:46,951 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,951 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:46,951 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:46,952 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:46,953 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:46,953 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:46,954 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:46,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:46,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:46,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:46,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:46,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:46,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:46,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24aa6e15[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:46,956 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:46,956 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:46,958 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:46,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:46,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:46,959 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:46,959 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:46,959 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:46,959 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,959 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:46,960 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:46,960 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:46,960 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:46,960 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:46,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:46,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:46,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:46,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:46,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:46,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:46,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:46,963 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:46,963 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:46,963 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:46,963 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:46,963 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:46,963 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:46,965 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:46,965 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:46,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:46,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:46,965 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:46,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:46,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:46,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:46,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:46,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:46,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:46,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:46,966 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:46,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:46,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:46,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,967 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:46,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:46,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,970 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,970 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:46,970 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:46,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:46,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:46,970 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:46,970 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:46,970 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:46,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:46,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:46,970 DEBUG [Thread-1715] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:46,970 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:46,971 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:46,970 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:46,971 DEBUG [Thread-1715] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:46,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:46,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:46,971 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:46,971 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:47,071 DEBUG [Thread-1715] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:47,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:47,073 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:47,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:47,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:47,074 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:47,074 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:47,074 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:47,074 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:47,075 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:47,075 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:47,077 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:47,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:47,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:47,077 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:47,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:47,077 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:47,077 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:47,077 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:47,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:47,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:47,078 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:47,078 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:47,078 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:47,078 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:47,078 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:47,078 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:47,078 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:47,078 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:47,078 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:47,079 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. started... 2023-04-09 09:54:47,079 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 026e485e431b415e1bb021c3d14de88b 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-09 09:54:47,079 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=A 2023-04-09 09:54:47,079 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:47,079 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=B 2023-04-09 09:54:47,079 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:47,079 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=C 2023-04-09 09:54:47,079 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:47,086 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=291 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/A/40e3c6055f294144bbd76a30e0f64eaa 2023-04-09 09:54:47,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 026e485e431b415e1bb021c3d14de88b 2023-04-09 09:54:47,121 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. as already flushing 2023-04-09 09:54:47,161 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681034087153, max timestamp=1681034087159Num uniques 859; 2023-04-09 09:54:47,161 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-09 09:54:47,161 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681034087146, max timestamp=1681034087159Num uniques 859; 2023-04-09 09:54:47,161 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681034087146, max timestamp=1681034087159Num uniques 859; 2023-04-09 09:54:47,161 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-09 09:54:47,161 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-09 09:54:47,177 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:47,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56138 deadline: 1681034147174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:47,177 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:47,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56106 deadline: 1681034147174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:47,177 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:47,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56098 deadline: 1681034147174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:47,177 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:47,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56126 deadline: 1681034147175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:47,211 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/C/db9800862dce40fdbc6a56ba303213a5 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/db9800862dce40fdbc6a56ba303213a5 2023-04-09 09:54:47,215 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 026e485e431b415e1bb021c3d14de88b/C of 026e485e431b415e1bb021c3d14de88b into db9800862dce40fdbc6a56ba303213a5(size=12.0 K), total size for store is 23.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:54:47,215 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 026e485e431b415e1bb021c3d14de88b: 2023-04-09 09:54:47,215 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b., storeName=026e485e431b415e1bb021c3d14de88b/C, priority=13, startTime=1681034086358; duration=0sec 2023-04-09 09:54:47,215 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:54:47,279 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:47,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56106 deadline: 1681034147278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:47,279 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:47,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56138 deadline: 1681034147278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:47,279 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:47,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56098 deadline: 1681034147278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:47,280 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:47,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56126 deadline: 1681034147278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:47,482 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:47,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56138 deadline: 1681034147481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:47,482 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:47,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56106 deadline: 1681034147481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:47,484 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:47,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56098 deadline: 1681034147482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:47,485 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:47,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56126 deadline: 1681034147482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:47,497 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=291 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/B/4b0abea584904282ae9d1ba6126ec9e1 2023-04-09 09:54:47,787 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:47,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56106 deadline: 1681034147785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:47,788 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:47,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56098 deadline: 1681034147785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:47,788 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:47,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56138 deadline: 1681034147786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:47,791 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:47,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56126 deadline: 1681034147788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:47,910 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=291 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/C/5dd892ec49314fce963be17b17273d46 2023-04-09 09:54:47,915 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/A/40e3c6055f294144bbd76a30e0f64eaa as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/40e3c6055f294144bbd76a30e0f64eaa 2023-04-09 09:54:47,921 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/40e3c6055f294144bbd76a30e0f64eaa, entries=150, sequenceid=291, filesize=11.9 K 2023-04-09 09:54:47,922 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/B/4b0abea584904282ae9d1ba6126ec9e1 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/4b0abea584904282ae9d1ba6126ec9e1 2023-04-09 09:54:47,925 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/4b0abea584904282ae9d1ba6126ec9e1, entries=150, sequenceid=291, filesize=11.9 K 2023-04-09 09:54:47,926 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/C/5dd892ec49314fce963be17b17273d46 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/5dd892ec49314fce963be17b17273d46 2023-04-09 09:54:47,931 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/5dd892ec49314fce963be17b17273d46, entries=150, sequenceid=291, filesize=11.9 K 2023-04-09 09:54:47,931 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=129.71 KB/132820 for 026e485e431b415e1bb021c3d14de88b in 852ms, sequenceid=291, compaction requested=true 2023-04-09 09:54:47,931 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 026e485e431b415e1bb021c3d14de88b: 2023-04-09 09:54:47,931 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:47,932 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:47,932 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:47,932 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:47,932 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:47,932 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:47,933 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:47,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:47,933 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:47,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:47,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:47,934 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:47,934 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:47,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:47,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:47,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:47,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:47,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:47,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:47,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70666fcc[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:47,935 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:47,935 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:47,936 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:47,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:47,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:47,936 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:47,936 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:47,936 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:47,937 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:47,937 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:47,938 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:47,938 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:47,938 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:47,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:47,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:47,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:47,938 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:47,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:47,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:47,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:47,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:47,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:47,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:47,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:47,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:47,940 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:47,940 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:47,940 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:47,940 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:47,940 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:47,940 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:47,943 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:47,943 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:47,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:47,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:47,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:47,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:47,943 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:47,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:47,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:47,943 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:47,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:47,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:47,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:47,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:47,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:47,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:47,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:47,945 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:47,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:47,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:47,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:47,948 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:47,948 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:47,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:47,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:47,948 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:47,948 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:47,948 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:47,948 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:47,948 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:47,948 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:47,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:47,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:47,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:47,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:47,948 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:47,948 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:47,948 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:47,948 DEBUG [Thread-1715] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:47,949 DEBUG [Thread-1715] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:48,049 DEBUG [Thread-1715] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:48,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:48,051 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:48,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:48,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:48,052 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:48,052 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:48,052 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:48,052 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:48,052 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:48,052 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:48,053 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:48,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:48,054 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:48,054 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:48,054 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:48,054 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:48,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:48,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:48,055 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:48,055 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:48,055 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:48,055 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:48,055 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:48,055 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:48,056 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:48,056 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:48,056 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:48,056 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. started... 2023-04-09 09:54:48,056 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 026e485e431b415e1bb021c3d14de88b 3/3 column families, dataSize=129.71 KB heapSize=360.77 KB 2023-04-09 09:54:48,056 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=A 2023-04-09 09:54:48,056 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:54:48,056 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=B 2023-04-09 09:54:48,056 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:54:48,056 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=C 2023-04-09 09:54:48,056 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:54:48,065 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=326 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/A/c4e1ed9274f740ccafcf844c36f81e2d 2023-04-09 09:54:48,076 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=326 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/B/e29f3b54a2264a4db673044816da1138 2023-04-09 09:54:48,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 026e485e431b415e1bb021c3d14de88b 2023-04-09 09:54:48,291 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. as already flushing 2023-04-09 09:54:48,301 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:48,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56106 deadline: 1681034148297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,301 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:48,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56126 deadline: 1681034148297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,305 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:48,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56138 deadline: 1681034148301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,306 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:48,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56098 deadline: 1681034148302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,405 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:48,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56106 deadline: 1681034148402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,405 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:48,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56126 deadline: 1681034148403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,408 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:48,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56138 deadline: 1681034148406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,408 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:48,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56098 deadline: 1681034148407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,495 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=326 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/C/0e13428080fc4e679a89c03840b75971 2023-04-09 09:54:48,500 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/A/c4e1ed9274f740ccafcf844c36f81e2d as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/c4e1ed9274f740ccafcf844c36f81e2d 2023-04-09 09:54:48,503 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/c4e1ed9274f740ccafcf844c36f81e2d, entries=150, sequenceid=326, filesize=11.9 K 2023-04-09 09:54:48,504 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/B/e29f3b54a2264a4db673044816da1138 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/e29f3b54a2264a4db673044816da1138 2023-04-09 09:54:48,508 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/e29f3b54a2264a4db673044816da1138, entries=150, sequenceid=326, filesize=11.9 K 2023-04-09 09:54:48,509 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/C/0e13428080fc4e679a89c03840b75971 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/0e13428080fc4e679a89c03840b75971 2023-04-09 09:54:48,512 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/0e13428080fc4e679a89c03840b75971, entries=150, sequenceid=326, filesize=11.9 K 2023-04-09 09:54:48,513 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~129.71 KB/132820, heapSize ~360.72 KB/369376, currentSize=60.38 KB/61830 for 026e485e431b415e1bb021c3d14de88b in 457ms, sequenceid=326, compaction requested=true 2023-04-09 09:54:48,513 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 026e485e431b415e1bb021c3d14de88b: 2023-04-09 09:54:48,513 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:48,513 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:48,513 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:48,513 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,513 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:48,513 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:48,515 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,515 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:48,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:48,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:48,515 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:48,515 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:48,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:48,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:48,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:48,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:48,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:48,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c3e703f[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:48,518 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:48,518 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:48,520 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:48,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:48,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:48,520 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:48,520 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:48,520 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:48,520 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,520 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:48,522 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,522 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:48,522 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:48,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,522 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:48,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:48,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:48,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:48,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:48,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:48,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:48,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:48,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:48,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:48,524 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:48,524 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:48,524 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:48,524 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:48,524 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:48,526 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:48,526 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:48,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:48,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:48,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:48,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:48,526 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:48,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:48,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:48,527 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:48,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:48,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:48,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:48,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:48,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:48,528 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:48,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:48,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,534 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:48,534 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,534 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:48,534 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:48,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:48,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:48,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:48,534 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:48,534 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:48,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:48,534 DEBUG [Thread-1715] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:48,534 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:48,535 DEBUG [Thread-1715] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:48,535 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,535 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:48,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:48,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:48,535 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:48,535 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:48,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 026e485e431b415e1bb021c3d14de88b 2023-04-09 09:54:48,608 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 026e485e431b415e1bb021c3d14de88b 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-09 09:54:48,609 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=A 2023-04-09 09:54:48,609 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:48,609 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=B 2023-04-09 09:54:48,609 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:48,609 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=C 2023-04-09 09:54:48,609 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:48,629 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:48,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56138 deadline: 1681034148625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,629 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:48,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56126 deadline: 1681034148627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,630 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=340 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/A/dc633494d4f54c7f961912c1c3641301 2023-04-09 09:54:48,643 DEBUG [Thread-1715] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:48,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:48,644 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:48,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56098 deadline: 1681034148628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,645 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:48,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56106 deadline: 1681034148629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,646 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:48,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:48,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:48,648 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:48,648 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:48,648 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:48,648 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:48,649 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:48,649 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:48,650 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,650 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,650 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:48,652 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:48,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:48,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:48,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:48,656 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:48,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:48,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:48,657 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:48,660 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:48,660 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:48,660 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:48,660 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:48,661 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:48,661 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:48,661 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:48,661 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. started... 2023-04-09 09:54:48,661 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. as already flushing 2023-04-09 09:54:48,661 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:48,661 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:48,661 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:48,661 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,661 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:48,661 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:48,663 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:48,663 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:48,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:48,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:48,664 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:48,664 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:48,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:48,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:48,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:48,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:48,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1075721[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:48,665 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:48,665 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:48,667 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:48,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:48,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:48,667 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:48,667 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:48,667 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:48,667 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,667 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:48,670 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,670 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:48,670 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:48,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:48,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:48,670 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:48,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:48,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:48,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:48,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:48,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:48,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:48,673 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:48,673 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:48,674 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:48,674 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:48,674 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:48,674 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:48,676 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:48,676 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:48,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:48,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:48,676 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:48,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:48,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:48,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:48,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:48,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:48,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:48,676 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:48,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:48,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:48,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:48,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,678 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:48,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:48,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,680 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:48,680 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:48,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:48,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:48,681 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,681 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:48,681 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:48,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:48,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:48,681 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:48,681 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,681 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:48,681 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:48,681 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:48,681 DEBUG [Thread-1715] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:48,681 DEBUG [Thread-1715] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:48,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:48,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:48,733 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:48,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56138 deadline: 1681034148731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,733 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:48,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56126 deadline: 1681034148731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,746 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:48,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56098 deadline: 1681034148746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,750 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:48,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56106 deadline: 1681034148747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,781 DEBUG [Thread-1715] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:48,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:48,784 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:48,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:48,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:48,785 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:48,785 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:48,785 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:48,785 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:48,786 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:48,786 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:48,787 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:48,787 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:48,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:48,787 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,787 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:48,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:48,788 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:48,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:48,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:48,788 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:48,800 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:48,800 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:48,800 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:48,800 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:48,800 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:48,803 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:48,803 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:48,803 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:48,803 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. started... 2023-04-09 09:54:48,803 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. as already flushing 2023-04-09 09:54:48,803 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:48,803 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:48,803 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:48,803 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,803 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:48,803 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:48,811 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:48,811 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:48,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:48,812 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:48,812 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:48,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:48,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:48,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:48,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:48,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:48,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76aa4bd2[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:48,814 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:48,814 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:48,815 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:48,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:48,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:48,815 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:48,815 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:48,815 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:48,815 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,815 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:48,816 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:48,816 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,816 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:48,817 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:48,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:48,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:48,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:48,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:48,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:48,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:48,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:48,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:48,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:48,819 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:48,819 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:48,819 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:48,819 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:48,819 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:48,820 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:48,820 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:48,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:48,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:48,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:48,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:48,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:48,820 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:48,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:48,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:48,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:48,821 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:48,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:48,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:48,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:48,822 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:48,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:48,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,825 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,825 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:48,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:48,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:48,825 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:48,826 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:48,826 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:48,826 DEBUG [Thread-1715] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:48,826 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:48,826 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,826 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:48,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:48,826 DEBUG [Thread-1715] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:48,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:48,826 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:48,826 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:48,826 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:48,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:48,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:48,926 DEBUG [Thread-1715] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:48,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:48,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:48,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:48,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:48,929 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:48,930 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:48,930 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:48,930 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:48,930 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:48,930 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:48,931 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,931 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:48,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:48,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:48,932 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,932 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:48,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:48,932 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:48,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:48,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:48,933 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:48,933 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:48,933 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:48,933 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:48,933 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:48,933 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:48,933 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:48,933 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:48,933 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:48,933 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. started... 2023-04-09 09:54:48,933 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. as already flushing 2023-04-09 09:54:48,934 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:48,934 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:48,934 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:48,934 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,934 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:48,934 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:48,935 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:48,935 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56126 deadline: 1681034148935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:48,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:48,935 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:48,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:48,936 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:48,936 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:48,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:48,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:48,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:48,937 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:48,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56138 deadline: 1681034148935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:48,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35f152c4[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:48,938 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:48,938 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:48,939 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:48,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:48,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:48,939 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:48,939 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,939 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:48,939 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:48,939 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:48,941 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,941 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:48,942 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:48,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:48,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:48,942 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:48,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:48,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:48,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:48,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:48,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:48,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:48,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:48,944 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:48,944 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:48,944 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:48,944 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:48,944 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:48,946 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:48,946 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:48,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:48,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:48,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:48,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:48,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:48,946 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:48,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:48,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:48,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:48,947 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:48,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:48,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:48,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:48,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,949 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:48,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:48,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,950 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:48,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56098 deadline: 1681034148947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,953 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:48,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56106 deadline: 1681034148951, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,953 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:48,954 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:48,954 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:48,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:48,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:48,954 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:48,954 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:48,954 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:48,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:48,954 DEBUG [Thread-1715] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:48,954 DEBUG [Thread-1715] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:48,954 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:48,954 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:48,954 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:48,954 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:48,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:48,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:49,054 DEBUG [Thread-1715] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:49,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:49,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:49,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:49,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:49,057 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:49,057 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:49,057 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:49,057 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:49,058 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:49,058 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:49,058 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=340 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/B/e6dcaf082c7846829bbf2ea146929c29 2023-04-09 09:54:49,063 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,063 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:49,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:49,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:49,064 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,064 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:49,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:49,064 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:49,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:49,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:49,064 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:49,065 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:49,065 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:49,065 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:49,065 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:49,065 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:49,065 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:49,065 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:49,065 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:49,065 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. started... 2023-04-09 09:54:49,065 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. as already flushing 2023-04-09 09:54:49,065 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:49,065 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:49,065 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:49,065 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,065 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:49,065 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:49,067 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,067 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:49,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:49,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:49,067 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:49,067 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:49,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:49,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:49,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:49,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:49,068 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:49,069 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:49,069 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:49,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a7163cd[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:49,070 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:49,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:49,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:49,070 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:49,070 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:49,070 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:49,070 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,070 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:49,072 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:49,072 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:49,072 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:49,072 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:49,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:49,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:49,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:49,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:49,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:49,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:49,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:49,075 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:49,075 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:49,075 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:49,075 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:49,075 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:49,075 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:49,077 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:49,077 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:49,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:49,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:49,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:49,077 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:49,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:49,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:49,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:49,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:49,078 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:49,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:49,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:49,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:49,079 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:49,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:49,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:49,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,082 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,082 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:49,082 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:49,082 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,082 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:49,082 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:49,082 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:49,082 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:49,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:49,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:49,082 DEBUG [Thread-1715] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:49,082 DEBUG [Thread-1715] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:49,084 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:49,084 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:49,084 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:49,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:49,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:49,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:49,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:49,182 DEBUG [Thread-1715] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:49,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:49,184 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:49,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:49,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:49,185 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:49,186 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:49,186 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:49,186 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:49,186 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:49,186 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:49,187 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:49,188 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:49,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:49,188 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,188 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:49,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:49,188 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:49,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:49,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:49,189 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:49,189 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:49,189 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:49,189 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:49,189 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:49,189 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:49,190 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:49,190 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:49,190 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:49,190 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. started... 2023-04-09 09:54:49,190 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. as already flushing 2023-04-09 09:54:49,190 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:49,190 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:49,190 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:49,190 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,190 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:49,190 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:49,192 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,192 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:49,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:49,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:49,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:49,192 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:49,193 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:49,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:49,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:49,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:49,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:49,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31f483fd[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:49,194 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:49,194 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:49,197 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:49,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:49,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:49,197 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:49,197 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:49,197 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:49,197 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,198 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:49,199 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,199 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:49,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:49,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:49,199 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:49,199 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:49,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:49,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:49,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:49,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:49,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:49,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:49,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:49,201 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:49,202 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:49,202 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:49,202 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:49,202 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:49,203 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:49,203 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:49,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:49,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:49,203 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:49,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:49,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:49,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:49,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:49,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:49,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:49,204 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:49,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:49,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:49,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:49,205 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:49,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:49,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,208 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:49,208 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,208 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:49,208 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:49,208 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:49,208 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,208 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:49,208 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:49,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:49,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:49,208 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:49,208 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:49,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:49,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:49,208 DEBUG [Thread-1715] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:49,208 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:49,208 DEBUG [Thread-1715] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:49,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:49,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:49,239 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:49,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56126 deadline: 1681034149238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,242 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:49,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56138 deadline: 1681034149240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,253 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:49,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56098 deadline: 1681034149252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,258 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:49,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56106 deadline: 1681034149256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,308 DEBUG [Thread-1715] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:49,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:49,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:49,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:49,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:49,316 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:49,316 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:49,316 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:49,316 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:49,317 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:49,317 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:49,320 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:49,320 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:49,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:49,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:49,321 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,321 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:49,321 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:49,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:49,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:49,322 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:49,328 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:49,328 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:49,328 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:49,328 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:49,328 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:49,328 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:49,328 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:49,328 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:49,328 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. started... 2023-04-09 09:54:49,329 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. as already flushing 2023-04-09 09:54:49,329 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:49,329 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:49,329 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:49,329 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,329 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:49,329 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:49,331 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:49,331 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:49,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:49,332 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:49,332 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:49,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:49,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:49,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:49,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:49,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:49,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1524aaca[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:49,333 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:49,333 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:49,334 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:49,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:49,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:49,335 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:49,335 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:49,335 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:49,335 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,335 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:49,336 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:49,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:49,337 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:49,337 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:49,337 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:49,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:49,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:49,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:49,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:49,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:49,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:49,344 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:49,344 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:49,344 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:49,344 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:49,344 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:49,344 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:49,347 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:49,347 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:49,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:49,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:49,347 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:49,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:49,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:49,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:49,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:49,348 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:49,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:49,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:49,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:49,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:49,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:49,351 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:49,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:49,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,356 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,357 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:49,357 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:49,357 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,357 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:49,357 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:49,357 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:49,357 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:49,357 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:49,357 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:49,357 DEBUG [Thread-1715] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:49,357 DEBUG [Thread-1715] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:49,357 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:49,357 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:49,357 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:49,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:49,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:49,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:49,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:49,457 DEBUG [Thread-1715] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:49,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:49,459 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:49,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:49,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:49,460 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:49,460 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:49,461 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:49,461 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:49,461 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:49,461 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:49,464 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:49,464 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:49,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:49,465 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:49,465 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:49,465 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:49,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:49,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:49,466 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:49,467 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:49,467 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:49,467 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:49,467 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:49,467 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:49,467 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:49,467 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:49,467 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:49,467 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. started... 2023-04-09 09:54:49,467 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. as already flushing 2023-04-09 09:54:49,468 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:49,468 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:49,468 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:49,468 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,468 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:49,468 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:49,470 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=340 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/C/5803cc5d47804bedaa8725d35cb8a647 2023-04-09 09:54:49,474 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/A/dc633494d4f54c7f961912c1c3641301 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/dc633494d4f54c7f961912c1c3641301 2023-04-09 09:54:49,478 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/dc633494d4f54c7f961912c1c3641301, entries=200, sequenceid=340, filesize=14.3 K 2023-04-09 09:54:49,479 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/B/e6dcaf082c7846829bbf2ea146929c29 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/e6dcaf082c7846829bbf2ea146929c29 2023-04-09 09:54:49,481 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,481 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:49,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:49,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:49,481 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:49,481 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:49,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:49,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:49,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:49,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:49,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:49,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2afcbdd8[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:49,483 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:49,483 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:49,484 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/e6dcaf082c7846829bbf2ea146929c29, entries=150, sequenceid=340, filesize=11.9 K 2023-04-09 09:54:49,484 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:49,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:49,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:49,485 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/C/5803cc5d47804bedaa8725d35cb8a647 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/5803cc5d47804bedaa8725d35cb8a647 2023-04-09 09:54:49,485 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,485 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:49,485 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:49,485 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:49,485 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:49,486 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,486 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:49,487 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:49,487 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:49,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:49,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:49,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:49,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:49,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:49,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:49,489 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/5803cc5d47804bedaa8725d35cb8a647, entries=150, sequenceid=340, filesize=11.9 K 2023-04-09 09:54:49,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:49,489 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 026e485e431b415e1bb021c3d14de88b in 881ms, sequenceid=340, compaction requested=true 2023-04-09 09:54:49,490 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 026e485e431b415e1bb021c3d14de88b: 2023-04-09 09:54:49,490 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:54:49,490 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-09 09:54:49,490 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-09 09:54:49,490 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:54:49,490 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:54:49,491 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 63494 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-09 09:54:49,491 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1912): 026e485e431b415e1bb021c3d14de88b/A is initiating minor compaction (all files) 2023-04-09 09:54:49,491 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 026e485e431b415e1bb021c3d14de88b/A in TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:49,492 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/c3866e6c1489427c9bc6d2f16462952a, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/b034770880d242788150337c0cc951c4, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/40e3c6055f294144bbd76a30e0f64eaa, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/c4e1ed9274f740ccafcf844c36f81e2d, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/dc633494d4f54c7f961912c1c3641301] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp, totalSize=62.0 K 2023-04-09 09:54:49,492 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61054 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-09 09:54:49,492 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): 026e485e431b415e1bb021c3d14de88b/B is initiating minor compaction (all files) 2023-04-09 09:54:49,492 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting c3866e6c1489427c9bc6d2f16462952a, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=242, earliestPutTs=1681034085284 2023-04-09 09:54:49,492 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 026e485e431b415e1bb021c3d14de88b/B in TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:49,492 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/6db091ddcac54a84a2b8fa9921b85016, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/713ce4b41b3348b78d88385a5fce2c4c, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/4b0abea584904282ae9d1ba6126ec9e1, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/e29f3b54a2264a4db673044816da1138, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/e6dcaf082c7846829bbf2ea146929c29] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp, totalSize=59.6 K 2023-04-09 09:54:49,492 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting b034770880d242788150337c0cc951c4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=278, earliestPutTs=1681034085994 2023-04-09 09:54:49,493 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 6db091ddcac54a84a2b8fa9921b85016, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=242, earliestPutTs=1681034085284 2023-04-09 09:54:49,493 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 40e3c6055f294144bbd76a30e0f64eaa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=291, earliestPutTs=1681034086537 2023-04-09 09:54:49,493 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 713ce4b41b3348b78d88385a5fce2c4c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=278, earliestPutTs=1681034085994 2023-04-09 09:54:49,493 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting c4e1ed9274f740ccafcf844c36f81e2d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=326, earliestPutTs=1681034087173 2023-04-09 09:54:49,493 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 4b0abea584904282ae9d1ba6126ec9e1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=291, earliestPutTs=1681034086537 2023-04-09 09:54:49,493 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting dc633494d4f54c7f961912c1c3641301, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=340, earliestPutTs=1681034088296 2023-04-09 09:54:49,494 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting e29f3b54a2264a4db673044816da1138, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=326, earliestPutTs=1681034087173 2023-04-09 09:54:49,494 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting e6dcaf082c7846829bbf2ea146929c29, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=340, earliestPutTs=1681034088296 2023-04-09 09:54:49,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:49,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:49,497 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:49,497 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:49,497 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:49,497 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:49,497 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:49,500 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:49,500 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:49,500 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:49,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:49,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:49,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:49,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:49,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:49,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:49,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:49,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:49,501 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:49,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:49,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:49,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:49,503 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:49,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:49,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,506 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:49,506 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:49,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:49,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:49,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:49,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:49,506 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:49,507 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:49,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:49,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:49,508 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): 026e485e431b415e1bb021c3d14de88b#B#compaction#255 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:54:49,508 DEBUG [Thread-1715] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:49,509 DEBUG [Thread-1715] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:49,506 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,509 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:49,509 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:49,509 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,508 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] throttle.PressureAwareThroughputController(145): 026e485e431b415e1bb021c3d14de88b#A#compaction#256 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:54:49,509 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:49,510 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:49,510 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:49,527 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/A/c3f60e7cb514480aaf7b6075ffee4bdb as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/c3f60e7cb514480aaf7b6075ffee4bdb 2023-04-09 09:54:49,531 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 026e485e431b415e1bb021c3d14de88b/A of 026e485e431b415e1bb021c3d14de88b into c3f60e7cb514480aaf7b6075ffee4bdb(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:54:49,531 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 026e485e431b415e1bb021c3d14de88b: 2023-04-09 09:54:49,531 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b., storeName=026e485e431b415e1bb021c3d14de88b/A, priority=11, startTime=1681034089490; duration=0sec 2023-04-09 09:54:49,531 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:54:49,531 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-09 09:54:49,532 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61054 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-09 09:54:49,532 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1912): 026e485e431b415e1bb021c3d14de88b/C is initiating minor compaction (all files) 2023-04-09 09:54:49,533 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 026e485e431b415e1bb021c3d14de88b/C in TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:49,533 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/db9800862dce40fdbc6a56ba303213a5, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/f38a2531c1144f3fae0c8a9ad08d16ed, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/5dd892ec49314fce963be17b17273d46, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/0e13428080fc4e679a89c03840b75971, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/5803cc5d47804bedaa8725d35cb8a647] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp, totalSize=59.6 K 2023-04-09 09:54:49,533 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting db9800862dce40fdbc6a56ba303213a5, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=242, earliestPutTs=1681034085284 2023-04-09 09:54:49,533 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting f38a2531c1144f3fae0c8a9ad08d16ed, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=278, earliestPutTs=1681034085994 2023-04-09 09:54:49,534 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 5dd892ec49314fce963be17b17273d46, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=291, earliestPutTs=1681034086537 2023-04-09 09:54:49,534 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 0e13428080fc4e679a89c03840b75971, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=326, earliestPutTs=1681034087173 2023-04-09 09:54:49,534 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 5803cc5d47804bedaa8725d35cb8a647, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=340, earliestPutTs=1681034088296 2023-04-09 09:54:49,542 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] throttle.PressureAwareThroughputController(145): 026e485e431b415e1bb021c3d14de88b#C#compaction#257 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:54:49,565 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/C/53aee86ae46a4d048925d35254c620be as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/53aee86ae46a4d048925d35254c620be 2023-04-09 09:54:49,572 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 026e485e431b415e1bb021c3d14de88b/C of 026e485e431b415e1bb021c3d14de88b into 53aee86ae46a4d048925d35254c620be(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:54:49,572 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 026e485e431b415e1bb021c3d14de88b: 2023-04-09 09:54:49,572 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b., storeName=026e485e431b415e1bb021c3d14de88b/C, priority=11, startTime=1681034089490; duration=0sec 2023-04-09 09:54:49,573 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:54:49,609 DEBUG [Thread-1715] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:49,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:49,611 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:49,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:49,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:49,612 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:49,612 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:49,612 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:49,612 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:49,613 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:49,613 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:49,614 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,614 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:49,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:49,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:49,615 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,615 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:49,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:49,615 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:49,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:49,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:49,616 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:49,616 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:49,616 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:49,616 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:49,616 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:49,616 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:49,628 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:49,628 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:49,628 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:49,628 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. started... 2023-04-09 09:54:49,628 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 026e485e431b415e1bb021c3d14de88b 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-09 09:54:49,629 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=A 2023-04-09 09:54:49,629 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:49,629 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=B 2023-04-09 09:54:49,629 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:49,629 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=C 2023-04-09 09:54:49,629 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:49,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 026e485e431b415e1bb021c3d14de88b 2023-04-09 09:54:49,743 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. as already flushing 2023-04-09 09:54:49,760 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:49,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56126 deadline: 1681034149757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,760 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:49,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56098 deadline: 1681034149758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,760 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:49,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56106 deadline: 1681034149759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,762 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:49,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56138 deadline: 1681034149760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,782 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:49,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56168 deadline: 1681034149781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,784 DEBUG [Thread-1709] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8147 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b., hostname=jenkins-hbase4.apache.org,42467,1681034019544, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-09 09:54:49,863 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:49,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56126 deadline: 1681034149861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,864 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:49,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56138 deadline: 1681034149863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:49,923 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/B/9bc9c819a95745f9a4716363ab4321a1 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/9bc9c819a95745f9a4716363ab4321a1 2023-04-09 09:54:49,927 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 026e485e431b415e1bb021c3d14de88b/B of 026e485e431b415e1bb021c3d14de88b into 9bc9c819a95745f9a4716363ab4321a1(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:54:49,927 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 026e485e431b415e1bb021c3d14de88b: 2023-04-09 09:54:49,927 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b., storeName=026e485e431b415e1bb021c3d14de88b/B, priority=11, startTime=1681034089490; duration=0sec 2023-04-09 09:54:49,927 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:54:50,057 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=360 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/A/bbea8e3cc6c04a988444e4e62cfd4423 2023-04-09 09:54:50,067 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=360 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/B/d3ac8699bc2940179c8d4f0f78d80a16 2023-04-09 09:54:50,067 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:50,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56126 deadline: 1681034150065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:50,069 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:50,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56138 deadline: 1681034150067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:50,079 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=360 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/C/98c33ee6a374474bbe44ee5c7e8b3151 2023-04-09 09:54:50,083 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/A/bbea8e3cc6c04a988444e4e62cfd4423 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/bbea8e3cc6c04a988444e4e62cfd4423 2023-04-09 09:54:50,086 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/bbea8e3cc6c04a988444e4e62cfd4423, entries=150, sequenceid=360, filesize=11.9 K 2023-04-09 09:54:50,087 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/B/d3ac8699bc2940179c8d4f0f78d80a16 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/d3ac8699bc2940179c8d4f0f78d80a16 2023-04-09 09:54:50,090 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/d3ac8699bc2940179c8d4f0f78d80a16, entries=150, sequenceid=360, filesize=11.9 K 2023-04-09 09:54:50,091 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/C/98c33ee6a374474bbe44ee5c7e8b3151 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/98c33ee6a374474bbe44ee5c7e8b3151 2023-04-09 09:54:50,094 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/98c33ee6a374474bbe44ee5c7e8b3151, entries=150, sequenceid=360, filesize=11.9 K 2023-04-09 09:54:50,095 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 026e485e431b415e1bb021c3d14de88b in 467ms, sequenceid=360, compaction requested=false 2023-04-09 09:54:50,095 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 026e485e431b415e1bb021c3d14de88b: 2023-04-09 09:54:50,095 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:50,095 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:50,095 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:50,095 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:50,095 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:50,095 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:50,097 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:50,097 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:50,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:50,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:50,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:50,097 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:50,097 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:50,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:50,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:50,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:50,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:50,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:50,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:50,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b12cc41[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:50,098 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:50,099 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:50,100 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:50,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:50,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:50,100 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:50,100 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:50,100 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:50,100 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:50,100 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:50,102 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:50,102 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:50,102 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:50,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:50,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:50,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:50,102 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:50,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:50,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:50,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:50,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:50,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:50,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:50,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:50,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:50,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:50,105 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:50,105 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:50,105 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:50,105 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:50,105 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:50,106 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:50,106 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:50,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:50,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:50,107 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:50,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:50,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:50,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:50,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:50,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:50,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:50,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:50,107 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:50,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:50,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:50,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:50,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:50,109 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:50,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:50,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:50,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:50,111 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:50,111 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:50,111 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:50,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:50,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:50,111 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:50,111 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:50,111 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:50,111 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:50,111 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:50,111 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:50,111 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:50,111 DEBUG [Thread-1715] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:50,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:50,111 DEBUG [Thread-1715] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:50,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:50,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:50,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:50,212 DEBUG [Thread-1715] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:50,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:50,214 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:50,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:50,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:50,215 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:50,215 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:50,215 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:50,215 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:50,216 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:50,216 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:50,217 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:50,217 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:50,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:50,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:50,217 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:50,217 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:50,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:50,218 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:50,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:50,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:50,218 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:50,219 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:50,219 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:50,219 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:50,219 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:50,219 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:50,219 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:50,219 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:50,219 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:50,219 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. started... 2023-04-09 09:54:50,219 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 026e485e431b415e1bb021c3d14de88b 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-09 09:54:50,219 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=A 2023-04-09 09:54:50,219 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:50,219 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=B 2023-04-09 09:54:50,220 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:50,220 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=C 2023-04-09 09:54:50,220 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:50,227 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=375 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/A/7fc5b1bac4fc450f89fffa73aef02658 2023-04-09 09:54:50,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 026e485e431b415e1bb021c3d14de88b 2023-04-09 09:54:50,372 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. as already flushing 2023-04-09 09:54:50,394 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:50,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56138 deadline: 1681034150392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:50,396 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:50,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56126 deadline: 1681034150394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:50,497 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:50,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56138 deadline: 1681034150495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:50,498 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:50,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56126 deadline: 1681034150497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:50,637 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=375 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/B/93235309e4db4d908ad350ad8ce89fff 2023-04-09 09:54:50,699 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:50,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56138 deadline: 1681034150698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:50,700 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:50,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56126 deadline: 1681034150699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:50,769 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:50,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56098 deadline: 1681034150768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:50,772 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:50,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56106 deadline: 1681034150770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:51,003 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:51,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56126 deadline: 1681034151002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:51,003 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:51,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56138 deadline: 1681034151002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:51,047 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=375 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/C/471df4ad1aa34cdbbca1049ebcb45cf1 2023-04-09 09:54:51,051 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/A/7fc5b1bac4fc450f89fffa73aef02658 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/7fc5b1bac4fc450f89fffa73aef02658 2023-04-09 09:54:51,055 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/7fc5b1bac4fc450f89fffa73aef02658, entries=150, sequenceid=375, filesize=11.9 K 2023-04-09 09:54:51,056 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/B/93235309e4db4d908ad350ad8ce89fff as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/93235309e4db4d908ad350ad8ce89fff 2023-04-09 09:54:51,059 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/93235309e4db4d908ad350ad8ce89fff, entries=150, sequenceid=375, filesize=11.9 K 2023-04-09 09:54:51,060 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/C/471df4ad1aa34cdbbca1049ebcb45cf1 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/471df4ad1aa34cdbbca1049ebcb45cf1 2023-04-09 09:54:51,063 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/471df4ad1aa34cdbbca1049ebcb45cf1, entries=150, sequenceid=375, filesize=11.9 K 2023-04-09 09:54:51,064 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 026e485e431b415e1bb021c3d14de88b in 845ms, sequenceid=375, compaction requested=true 2023-04-09 09:54:51,064 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 026e485e431b415e1bb021c3d14de88b: 2023-04-09 09:54:51,064 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:51,064 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:51,064 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:51,064 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:51,064 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:51,064 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:51,066 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:51,066 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:51,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:51,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:51,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:51,067 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:51,067 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:51,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:51,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:51,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:51,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:51,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:51,068 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:51,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e6c49b5[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:51,069 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:51,069 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:51,070 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:51,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:51,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:51,070 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:51,070 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:51,070 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:51,071 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:51,071 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:51,072 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:51,072 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:51,072 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:51,072 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:51,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:51,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:51,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:51,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:51,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:51,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:51,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:51,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:51,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:51,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:51,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:51,075 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:51,075 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:51,075 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:51,075 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:51,075 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:51,075 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:51,078 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:51,078 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:51,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:51,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:51,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:51,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:51,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:51,078 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:51,078 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:51,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:51,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:51,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:51,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:51,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:51,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:51,079 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:51,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:51,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:51,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:51,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:51,081 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:51,081 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:51,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:51,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:51,082 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:51,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:51,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:51,082 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:51,082 DEBUG [Thread-1715] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:51,081 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:51,082 DEBUG [Thread-1715] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:51,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:51,082 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:51,082 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:51,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:51,082 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:51,082 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:51,082 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:51,082 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:51,182 DEBUG [Thread-1715] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:51,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:51,184 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:51,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:51,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:51,186 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:51,186 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:51,186 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:51,186 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:51,186 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:51,186 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:51,187 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:51,187 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:51,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:51,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:51,188 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:51,188 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:51,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:51,188 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:51,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:51,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:51,189 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:51,189 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:51,189 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:51,189 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:51,189 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:51,189 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:51,189 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:51,189 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:51,189 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:51,189 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. started... 2023-04-09 09:54:51,190 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 026e485e431b415e1bb021c3d14de88b 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-09 09:54:51,190 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=A 2023-04-09 09:54:51,190 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:51,190 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=B 2023-04-09 09:54:51,190 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:51,190 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=C 2023-04-09 09:54:51,190 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:51,197 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=393 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/A/681601aab7644da79a4f1a9bd5d3e798 2023-04-09 09:54:51,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 026e485e431b415e1bb021c3d14de88b 2023-04-09 09:54:51,507 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. as already flushing 2023-04-09 09:54:51,523 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:51,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56138 deadline: 1681034151521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:51,524 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:51,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56126 deadline: 1681034151522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:51,626 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=393 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/B/8dd1a7e0543b4e3da9c5b8227f58a6c7 2023-04-09 09:54:51,627 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:51,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56138 deadline: 1681034151624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:51,628 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:51,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56126 deadline: 1681034151625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:51,830 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:51,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56138 deadline: 1681034151828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:51,831 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:51,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56126 deadline: 1681034151829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,035 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=393 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/C/ca5b2f2457f94ab2be06985611429bb1 2023-04-09 09:54:52,040 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/A/681601aab7644da79a4f1a9bd5d3e798 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/681601aab7644da79a4f1a9bd5d3e798 2023-04-09 09:54:52,043 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/681601aab7644da79a4f1a9bd5d3e798, entries=150, sequenceid=393, filesize=11.9 K 2023-04-09 09:54:52,044 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/B/8dd1a7e0543b4e3da9c5b8227f58a6c7 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/8dd1a7e0543b4e3da9c5b8227f58a6c7 2023-04-09 09:54:52,047 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/8dd1a7e0543b4e3da9c5b8227f58a6c7, entries=150, sequenceid=393, filesize=11.9 K 2023-04-09 09:54:52,048 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/C/ca5b2f2457f94ab2be06985611429bb1 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/ca5b2f2457f94ab2be06985611429bb1 2023-04-09 09:54:52,051 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/ca5b2f2457f94ab2be06985611429bb1, entries=150, sequenceid=393, filesize=11.9 K 2023-04-09 09:54:52,052 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 026e485e431b415e1bb021c3d14de88b in 862ms, sequenceid=393, compaction requested=true 2023-04-09 09:54:52,052 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 026e485e431b415e1bb021c3d14de88b: 2023-04-09 09:54:52,052 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:52,052 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:52,052 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:52,052 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,052 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:52,052 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:52,056 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,056 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:52,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:52,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:52,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:52,057 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:52,057 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:52,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:52,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:52,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:52,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:52,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e2ed648[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:52,058 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:52,058 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:52,061 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:52,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:52,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:52,061 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:52,061 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:52,061 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,061 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:52,061 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:52,063 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:52,063 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:52,063 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,063 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:52,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:52,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:52,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:52,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:52,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:52,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:52,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:52,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:52,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:52,066 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:52,066 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:52,066 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:52,066 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:52,066 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:52,068 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:52,068 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:52,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:52,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:52,069 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:52,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:52,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:52,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:52,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:52,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:52,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:52,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:52,069 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:52,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:52,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:52,070 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:52,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:52,072 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,072 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:52,072 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:52,072 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:52,073 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:52,072 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:52,072 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:52,073 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:52,073 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:52,073 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:52,073 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:52,073 DEBUG [Thread-1715] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:52,073 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:52,073 DEBUG [Thread-1715] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:52,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:52,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:52,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:52,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:52,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 026e485e431b415e1bb021c3d14de88b 2023-04-09 09:54:52,135 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 026e485e431b415e1bb021c3d14de88b 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-09 09:54:52,135 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=A 2023-04-09 09:54:52,135 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:52,135 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=B 2023-04-09 09:54:52,135 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:52,135 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=C 2023-04-09 09:54:52,135 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:52,154 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=408 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/A/58847ab295994ed5b8b805376c7dee2e 2023-04-09 09:54:52,169 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=408 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/B/f5cb4f45579a4c9289d52bc94c37b8cd 2023-04-09 09:54:52,170 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:52,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56126 deadline: 1681034152166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,170 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:52,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56138 deadline: 1681034152167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,173 DEBUG [Thread-1715] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:52,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:52,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:52,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:52,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:52,180 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:52,180 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:52,180 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:52,180 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:52,180 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:52,180 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:52,183 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:52,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:52,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:52,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:52,187 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:52,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:52,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:52,188 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:52,188 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,188 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,188 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:52,200 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:52,200 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:52,200 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:52,200 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:52,200 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:52,208 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:52,208 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:52,208 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:52,208 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. started... 2023-04-09 09:54:52,208 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. as already flushing 2023-04-09 09:54:52,208 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:52,208 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:52,208 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:52,208 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,220 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:52,220 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:52,222 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,222 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:52,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:52,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:52,222 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:52,222 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:52,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:52,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:52,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:52,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:52,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:52,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4173a828[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:52,224 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:52,225 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:52,226 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:52,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:52,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:52,226 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:52,226 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:52,226 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:52,226 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,226 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:52,227 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,227 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:52,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:52,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:52,227 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:52,228 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:52,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:52,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:52,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:52,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:52,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:52,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,231 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=408 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/C/1746922aba2f49a2bb62fd8ed5c97454 2023-04-09 09:54:52,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:52,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:52,231 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:52,231 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:52,231 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:52,231 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:52,231 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:52,236 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:52,236 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:52,236 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:52,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:52,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:52,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:52,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:52,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:52,236 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/A/58847ab295994ed5b8b805376c7dee2e as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/58847ab295994ed5b8b805376c7dee2e 2023-04-09 09:54:52,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:52,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:52,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:52,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:52,237 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:52,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:52,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:52,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:52,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:52,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,246 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/58847ab295994ed5b8b805376c7dee2e, entries=200, sequenceid=408, filesize=14.3 K 2023-04-09 09:54:52,247 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,247 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/B/f5cb4f45579a4c9289d52bc94c37b8cd as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/f5cb4f45579a4c9289d52bc94c37b8cd 2023-04-09 09:54:52,250 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,250 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:52,250 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:52,250 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:52,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:52,250 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:52,250 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:52,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:52,250 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:52,250 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:52,250 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,250 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:52,250 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:52,250 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:52,250 DEBUG [Thread-1715] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:52,250 DEBUG [Thread-1715] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:52,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:52,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:52,254 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/f5cb4f45579a4c9289d52bc94c37b8cd, entries=150, sequenceid=408, filesize=11.9 K 2023-04-09 09:54:52,255 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/C/1746922aba2f49a2bb62fd8ed5c97454 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/1746922aba2f49a2bb62fd8ed5c97454 2023-04-09 09:54:52,258 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/1746922aba2f49a2bb62fd8ed5c97454, entries=150, sequenceid=408, filesize=11.9 K 2023-04-09 09:54:52,259 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 026e485e431b415e1bb021c3d14de88b in 124ms, sequenceid=408, compaction requested=true 2023-04-09 09:54:52,259 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 026e485e431b415e1bb021c3d14de88b: 2023-04-09 09:54:52,259 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:54:52,259 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-09 09:54:52,259 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-09 09:54:52,259 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:54:52,260 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:54:52,261 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 63814 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-09 09:54:52,261 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61374 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-09 09:54:52,261 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1912): 026e485e431b415e1bb021c3d14de88b/A is initiating minor compaction (all files) 2023-04-09 09:54:52,261 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): 026e485e431b415e1bb021c3d14de88b/B is initiating minor compaction (all files) 2023-04-09 09:54:52,261 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 026e485e431b415e1bb021c3d14de88b/B in TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:52,261 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 026e485e431b415e1bb021c3d14de88b/A in TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:52,261 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/9bc9c819a95745f9a4716363ab4321a1, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/d3ac8699bc2940179c8d4f0f78d80a16, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/93235309e4db4d908ad350ad8ce89fff, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/8dd1a7e0543b4e3da9c5b8227f58a6c7, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/f5cb4f45579a4c9289d52bc94c37b8cd] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp, totalSize=59.9 K 2023-04-09 09:54:52,261 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/c3f60e7cb514480aaf7b6075ffee4bdb, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/bbea8e3cc6c04a988444e4e62cfd4423, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/7fc5b1bac4fc450f89fffa73aef02658, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/681601aab7644da79a4f1a9bd5d3e798, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/58847ab295994ed5b8b805376c7dee2e] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp, totalSize=62.3 K 2023-04-09 09:54:52,261 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 9bc9c819a95745f9a4716363ab4321a1, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=340, earliestPutTs=1681034088296 2023-04-09 09:54:52,262 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting c3f60e7cb514480aaf7b6075ffee4bdb, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=340, earliestPutTs=1681034088296 2023-04-09 09:54:52,262 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting d3ac8699bc2940179c8d4f0f78d80a16, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=360, earliestPutTs=1681034088624 2023-04-09 09:54:52,262 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting bbea8e3cc6c04a988444e4e62cfd4423, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=360, earliestPutTs=1681034088624 2023-04-09 09:54:52,262 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 93235309e4db4d908ad350ad8ce89fff, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=375, earliestPutTs=1681034089752 2023-04-09 09:54:52,263 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 7fc5b1bac4fc450f89fffa73aef02658, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=375, earliestPutTs=1681034089752 2023-04-09 09:54:52,263 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 8dd1a7e0543b4e3da9c5b8227f58a6c7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=393, earliestPutTs=1681034090390 2023-04-09 09:54:52,263 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 681601aab7644da79a4f1a9bd5d3e798, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=393, earliestPutTs=1681034090390 2023-04-09 09:54:52,263 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting f5cb4f45579a4c9289d52bc94c37b8cd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=408, earliestPutTs=1681034091511 2023-04-09 09:54:52,263 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 58847ab295994ed5b8b805376c7dee2e, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=408, earliestPutTs=1681034091511 2023-04-09 09:54:52,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 026e485e431b415e1bb021c3d14de88b 2023-04-09 09:54:52,275 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 026e485e431b415e1bb021c3d14de88b 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-09 09:54:52,276 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=A 2023-04-09 09:54:52,276 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:52,276 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=B 2023-04-09 09:54:52,276 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:52,276 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=C 2023-04-09 09:54:52,276 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:52,282 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): 026e485e431b415e1bb021c3d14de88b#B#compaction#271 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:54:52,286 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] throttle.PressureAwareThroughputController(145): 026e485e431b415e1bb021c3d14de88b#A#compaction#272 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:54:52,290 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=426 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/A/ea78ac69f98e4f8e84169e39aad7ba50 2023-04-09 09:54:52,304 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:52,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56138 deadline: 1681034152300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,305 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:52,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56126 deadline: 1681034152305, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,309 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=426 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/B/abcfc87f65874425a6dfef508f0e91a5 2023-04-09 09:54:52,350 DEBUG [Thread-1715] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:52,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:52,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:52,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:52,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:52,354 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:52,354 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:52,354 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:52,354 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:52,354 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:52,354 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:52,355 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,355 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:52,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:52,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:52,356 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,356 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:52,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:52,356 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:52,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:52,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:52,357 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:52,357 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:52,357 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:52,357 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:52,357 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:52,357 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:52,357 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:52,357 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:52,357 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:52,357 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. started... 2023-04-09 09:54:52,357 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. as already flushing 2023-04-09 09:54:52,358 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:52,358 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:52,358 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:52,358 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,358 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:52,358 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:52,359 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:52,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:52,359 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:52,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:52,359 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:52,359 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:52,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:52,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:52,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:52,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:52,361 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7cbe0483[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:52,361 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:52,361 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:52,368 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:52,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:52,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:52,368 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:52,368 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:52,368 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:52,369 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,369 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:52,370 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:52,370 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:52,370 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:52,370 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:52,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:52,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:52,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:52,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:52,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:52,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:52,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:52,372 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:52,372 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:52,372 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:52,372 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:52,372 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:52,372 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:52,373 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:52,373 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:52,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:52,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:52,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:52,374 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:52,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:52,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:52,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:52,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:52,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:52,374 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:52,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:52,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:52,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:52,376 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:52,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:52,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,378 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,378 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:52,378 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:52,378 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:52,378 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:52,378 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:52,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:52,378 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:52,378 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:52,378 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:52,378 DEBUG [Thread-1715] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:52,378 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:52,378 DEBUG [Thread-1715] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:52,378 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:52,378 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:52,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:52,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:52,407 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:52,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56138 deadline: 1681034152406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,407 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:52,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56126 deadline: 1681034152406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,478 DEBUG [Thread-1715] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:52,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:52,481 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:52,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:52,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:52,482 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:52,482 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:52,482 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:52,482 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:52,483 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:52,483 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:52,485 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:52,485 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:52,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:52,485 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:52,485 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:52,486 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:52,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:52,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:52,486 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:52,486 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:52,486 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:52,487 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:52,487 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:52,487 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:52,487 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:52,487 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:52,487 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:52,487 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. started... 2023-04-09 09:54:52,487 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. as already flushing 2023-04-09 09:54:52,487 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:52,487 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:52,487 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:52,487 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,487 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:52,487 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:52,489 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,489 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:52,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:52,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:52,489 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:52,489 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:52,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:52,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:52,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:52,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:52,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:52,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2043cb40[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:52,491 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:52,491 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:52,492 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:52,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:52,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:52,492 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:52,492 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:52,492 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:52,492 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,492 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:52,498 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:52,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:52,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:52,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:52,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:52,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:52,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:52,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,500 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:52,500 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:52,500 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:52,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:52,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:52,500 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:52,500 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:52,500 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:52,501 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:52,501 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:52,502 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:52,502 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:52,502 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:52,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:52,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:52,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:52,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:52,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:52,503 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:52,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:52,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:52,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:52,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:52,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:52,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:52,504 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:52,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:52,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,506 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:52,506 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:52,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:52,507 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:52,507 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:52,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:52,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:52,507 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:52,507 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:52,507 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,507 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:52,507 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:52,507 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:52,507 DEBUG [Thread-1715] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:52,507 DEBUG [Thread-1715] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:52,507 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:52,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:52,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:52,607 DEBUG [Thread-1715] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:52,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:52,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:52,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:52,610 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:52,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:52,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56126 deadline: 1681034152608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,611 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:52,611 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:52,611 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:52,611 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:52,611 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:52,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56138 deadline: 1681034152609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,611 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:52,611 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:52,613 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:52,613 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:52,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:52,613 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,613 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:52,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:52,613 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:52,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:52,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:52,614 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:52,614 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:52,614 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:52,614 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:52,614 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:52,614 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:52,614 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:52,614 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:52,615 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:52,615 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. started... 2023-04-09 09:54:52,615 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. as already flushing 2023-04-09 09:54:52,615 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:52,615 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:52,615 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:52,615 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,615 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:52,615 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:52,617 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,617 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:52,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:52,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:52,617 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:52,617 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:52,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:52,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:52,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:52,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:52,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:52,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5827d596[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:52,619 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:52,619 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:52,620 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:52,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:52,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:52,620 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:52,620 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:52,620 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:52,620 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,620 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:52,621 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,621 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:52,621 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:52,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,621 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:52,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:52,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:52,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:52,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:52,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:52,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:52,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:52,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:52,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:52,624 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:52,624 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:52,624 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:52,624 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:52,624 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:52,626 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:52,626 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:52,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:52,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:52,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:52,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:52,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:52,626 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:52,626 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:52,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:52,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:52,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:52,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:52,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:52,627 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:52,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:52,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:52,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,630 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,630 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:52,630 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:52,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:52,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:52,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:52,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:52,630 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:52,630 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:52,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:52,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:52,631 DEBUG [Thread-1715] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:52,631 DEBUG [Thread-1715] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:52,631 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:52,630 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:52,631 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,631 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:52,631 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:52,631 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:52,700 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/B/ed8f2fbee1784e998f9951b81f43ef2b as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/ed8f2fbee1784e998f9951b81f43ef2b 2023-04-09 09:54:52,704 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 026e485e431b415e1bb021c3d14de88b/B of 026e485e431b415e1bb021c3d14de88b into ed8f2fbee1784e998f9951b81f43ef2b(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:54:52,704 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 026e485e431b415e1bb021c3d14de88b: 2023-04-09 09:54:52,704 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b., storeName=026e485e431b415e1bb021c3d14de88b/B, priority=11, startTime=1681034092259; duration=0sec 2023-04-09 09:54:52,704 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:54:52,704 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-09 09:54:52,706 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/A/5e607fee10024a0580f41cd46797a436 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/5e607fee10024a0580f41cd46797a436 2023-04-09 09:54:52,706 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61374 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-09 09:54:52,706 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): 026e485e431b415e1bb021c3d14de88b/C is initiating minor compaction (all files) 2023-04-09 09:54:52,706 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 026e485e431b415e1bb021c3d14de88b/C in TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:52,706 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/53aee86ae46a4d048925d35254c620be, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/98c33ee6a374474bbe44ee5c7e8b3151, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/471df4ad1aa34cdbbca1049ebcb45cf1, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/ca5b2f2457f94ab2be06985611429bb1, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/1746922aba2f49a2bb62fd8ed5c97454] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp, totalSize=59.9 K 2023-04-09 09:54:52,706 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 53aee86ae46a4d048925d35254c620be, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=340, earliestPutTs=1681034088296 2023-04-09 09:54:52,707 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 98c33ee6a374474bbe44ee5c7e8b3151, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=360, earliestPutTs=1681034088624 2023-04-09 09:54:52,708 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 471df4ad1aa34cdbbca1049ebcb45cf1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=375, earliestPutTs=1681034089752 2023-04-09 09:54:52,709 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting ca5b2f2457f94ab2be06985611429bb1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=393, earliestPutTs=1681034090390 2023-04-09 09:54:52,709 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 1746922aba2f49a2bb62fd8ed5c97454, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=408, earliestPutTs=1681034091511 2023-04-09 09:54:52,711 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 026e485e431b415e1bb021c3d14de88b/A of 026e485e431b415e1bb021c3d14de88b into 5e607fee10024a0580f41cd46797a436(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:54:52,711 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 026e485e431b415e1bb021c3d14de88b: 2023-04-09 09:54:52,711 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b., storeName=026e485e431b415e1bb021c3d14de88b/A, priority=11, startTime=1681034092259; duration=0sec 2023-04-09 09:54:52,711 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:54:52,719 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): 026e485e431b415e1bb021c3d14de88b#C#compaction#275 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:54:52,722 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=426 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/C/700fc3f06f134c4e8c192323af609ac0 2023-04-09 09:54:52,727 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/A/ea78ac69f98e4f8e84169e39aad7ba50 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/ea78ac69f98e4f8e84169e39aad7ba50 2023-04-09 09:54:52,727 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/C/2052ff9828184c84aa0abba422b4fcbe as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/2052ff9828184c84aa0abba422b4fcbe 2023-04-09 09:54:52,731 DEBUG [Thread-1715] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:52,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:52,732 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/ea78ac69f98e4f8e84169e39aad7ba50, entries=150, sequenceid=426, filesize=11.9 K 2023-04-09 09:54:52,733 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/B/abcfc87f65874425a6dfef508f0e91a5 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/abcfc87f65874425a6dfef508f0e91a5 2023-04-09 09:54:52,734 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:52,734 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 026e485e431b415e1bb021c3d14de88b/C of 026e485e431b415e1bb021c3d14de88b into 2052ff9828184c84aa0abba422b4fcbe(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:54:52,734 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 026e485e431b415e1bb021c3d14de88b: 2023-04-09 09:54:52,734 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b., storeName=026e485e431b415e1bb021c3d14de88b/C, priority=11, startTime=1681034092260; duration=0sec 2023-04-09 09:54:52,734 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:54:52,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:52,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:52,735 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:52,735 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:52,735 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:52,735 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:52,735 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:52,735 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:52,736 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:52,736 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:52,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:52,737 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,737 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:52,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:52,738 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/abcfc87f65874425a6dfef508f0e91a5, entries=150, sequenceid=426, filesize=11.9 K 2023-04-09 09:54:52,739 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/C/700fc3f06f134c4e8c192323af609ac0 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/700fc3f06f134c4e8c192323af609ac0 2023-04-09 09:54:52,742 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:52,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:52,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:52,742 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:52,742 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/700fc3f06f134c4e8c192323af609ac0, entries=150, sequenceid=426, filesize=11.9 K 2023-04-09 09:54:52,742 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:52,743 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:52,743 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:52,743 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:52,743 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:52,743 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:52,743 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:52,743 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:52,743 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. started... 2023-04-09 09:54:52,743 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. as already flushing 2023-04-09 09:54:52,743 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:52,743 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:52,743 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:52,744 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,744 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:52,744 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:52,745 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,745 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:52,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:52,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:52,749 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 026e485e431b415e1bb021c3d14de88b in 474ms, sequenceid=426, compaction requested=false 2023-04-09 09:54:52,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:52,749 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 026e485e431b415e1bb021c3d14de88b: 2023-04-09 09:54:52,749 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:52,749 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:52,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:52,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:52,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:52,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:52,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4490452[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:52,750 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:52,750 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:52,752 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:52,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:52,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:52,752 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:52,752 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,752 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:52,752 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:52,752 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:52,754 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,754 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:52,754 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:52,754 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:52,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:52,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:52,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:52,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:52,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:52,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:52,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:52,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:52,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:52,758 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:52,758 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:52,758 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:52,759 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:52,759 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:52,760 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:52,760 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:52,760 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:52,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:52,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:52,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:52,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:52,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:52,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:52,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:52,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:52,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:52,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:52,763 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:52,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:52,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:52,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:52,766 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,770 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:52,770 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:52,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:52,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:52,770 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,770 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:52,770 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:52,770 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,770 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:52,771 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:52,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:52,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:52,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:52,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:52,771 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:52,771 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:52,771 DEBUG [Thread-1715] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:52,771 DEBUG [Thread-1715] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:52,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 026e485e431b415e1bb021c3d14de88b 2023-04-09 09:54:52,785 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 026e485e431b415e1bb021c3d14de88b 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-09 09:54:52,785 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=A 2023-04-09 09:54:52,785 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:52,785 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=B 2023-04-09 09:54:52,785 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:52,785 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=C 2023-04-09 09:54:52,785 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:52,795 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=444 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/A/6fd0c9634496420b87fa599434913b92 2023-04-09 09:54:52,819 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:52,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56098 deadline: 1681034152818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,821 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:52,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56106 deadline: 1681034152819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,871 DEBUG [Thread-1715] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:52,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:52,874 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:52,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:52,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:52,875 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:52,875 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:52,875 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:52,875 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:52,875 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:52,875 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:52,876 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,877 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:52,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:52,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:52,877 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,877 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:52,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:52,877 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:52,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:52,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:52,878 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:52,878 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:52,878 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:52,878 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:52,878 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:52,878 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:52,878 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:52,878 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:52,878 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:52,879 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. started... 2023-04-09 09:54:52,879 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. as already flushing 2023-04-09 09:54:52,879 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:52,879 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:52,879 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:52,879 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,879 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:52,879 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:52,882 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:52,882 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:52,882 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:52,882 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:52,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:52,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:52,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:52,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:52,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:52,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:52,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@85f8244[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:52,884 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:52,884 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:52,885 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:52,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:52,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:52,885 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:52,885 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:52,885 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:52,886 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,886 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:52,887 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,887 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:52,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:52,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:52,887 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:52,887 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:52,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:52,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:52,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:52,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:52,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:52,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:52,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:52,890 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:52,890 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:52,890 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:52,890 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:52,890 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:52,891 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:52,892 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:52,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:52,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:52,892 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:52,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:52,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:52,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:52,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:52,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:52,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:52,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:52,893 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:52,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:52,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:52,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,894 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:52,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:52,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,904 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,905 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:52,905 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:52,905 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:52,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:52,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:52,905 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:52,905 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:52,905 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:52,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:52,905 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:52,905 DEBUG [Thread-1715] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:52,905 DEBUG [Thread-1715] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:52,905 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:52,905 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:52,905 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:52,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:52,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:52,912 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:52,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56126 deadline: 1681034152912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,913 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:52,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56138 deadline: 1681034152912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,921 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:52,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56098 deadline: 1681034152920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:52,923 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:52,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56106 deadline: 1681034152922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:53,005 DEBUG [Thread-1715] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:53,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:53,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:53,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:53,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:53,009 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:53,009 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:53,009 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:53,009 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:53,009 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:53,009 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:53,010 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:53,010 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:53,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:53,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:53,011 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:53,011 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:53,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:53,011 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:53,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:53,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:53,012 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:53,012 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:53,012 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:53,012 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:53,012 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:53,012 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:53,012 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:53,012 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:53,012 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:53,012 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. started... 2023-04-09 09:54:53,012 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. as already flushing 2023-04-09 09:54:53,013 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:53,013 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:53,013 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:53,013 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:53,013 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:53,013 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:53,014 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:53,014 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:53,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:53,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:53,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:53,014 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:53,014 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:53,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:53,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:53,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:53,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:53,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:53,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:53,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36f841c4[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:53,016 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:53,016 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:53,017 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:53,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:53,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:53,017 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:53,017 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:53,017 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:53,017 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:53,017 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:53,019 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:53,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:53,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:53,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:53,020 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:53,020 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:53,020 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:53,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:53,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:53,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:53,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:53,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:53,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:53,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:53,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:53,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:53,022 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:53,022 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:53,022 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:53,022 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:53,022 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:53,024 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:53,024 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:53,024 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:53,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:53,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:53,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:53,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:53,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:53,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:53,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:53,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:53,025 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:53,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:53,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:53,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:53,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:53,026 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:53,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:53,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:53,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:53,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:53,029 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:53,029 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:53,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:53,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:53,029 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:53,029 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:53,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:53,029 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:53,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:53,029 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:53,029 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:53,029 DEBUG [Thread-1715] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:53,030 DEBUG [Thread-1715] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:53,029 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:53,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:53,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:53,030 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:53,030 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:53,030 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:53,123 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:53,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56098 deadline: 1681034153122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:53,125 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:53,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56106 deadline: 1681034153124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:53,130 DEBUG [Thread-1715] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:53,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:53,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:53,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:53,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:53,133 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:53,133 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:53,133 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:53,133 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:53,133 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:53,133 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:53,135 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:53,135 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:53,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:53,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:53,135 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:53,135 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:53,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:53,135 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:53,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:53,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:53,136 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:53,136 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:53,136 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:53,136 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:53,136 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:53,136 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:53,136 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:53,136 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:53,136 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:53,136 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. started... 2023-04-09 09:54:53,136 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. as already flushing 2023-04-09 09:54:53,137 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:53,137 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:53,137 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:53,137 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:53,137 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:53,137 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:53,139 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:53,139 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:53,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:53,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:53,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:53,139 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:53,139 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:53,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:53,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:53,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:53,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:53,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:53,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:53,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5015a9e8[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:53,140 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:53,140 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:53,141 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:53,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:53,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:53,142 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:53,142 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:53,142 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:53,142 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:53,142 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:53,143 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:53,143 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:53,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:53,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:53,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:53,143 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:53,143 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:53,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:53,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:53,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:53,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:53,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:53,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:53,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:53,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:53,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:53,145 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:53,145 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:53,145 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:53,145 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:53,145 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:53,147 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:53,147 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:53,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:53,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:53,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:53,147 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:53,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:53,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:53,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:53,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:53,147 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:53,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:53,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:53,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:53,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:53,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:53,149 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:53,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:53,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:53,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:53,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:53,152 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:53,152 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:53,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:53,152 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:53,152 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:53,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:53,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:53,152 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:53,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:53,152 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:53,152 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:53,152 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:53,152 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:53,152 DEBUG [Thread-1715] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:53,152 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:53,152 DEBUG [Thread-1715] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:53,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:53,152 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:53,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:53,218 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=444 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/B/4a1492c05b564fc5a8f95c0d2bad988a 2023-04-09 09:54:53,231 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=444 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/C/2fc52d3e940c430e829135e1b811783f 2023-04-09 09:54:53,234 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/A/6fd0c9634496420b87fa599434913b92 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/6fd0c9634496420b87fa599434913b92 2023-04-09 09:54:53,239 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/6fd0c9634496420b87fa599434913b92, entries=200, sequenceid=444, filesize=14.3 K 2023-04-09 09:54:53,239 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/B/4a1492c05b564fc5a8f95c0d2bad988a as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/4a1492c05b564fc5a8f95c0d2bad988a 2023-04-09 09:54:53,243 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/4a1492c05b564fc5a8f95c0d2bad988a, entries=150, sequenceid=444, filesize=11.9 K 2023-04-09 09:54:53,243 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/C/2fc52d3e940c430e829135e1b811783f as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/2fc52d3e940c430e829135e1b811783f 2023-04-09 09:54:53,246 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/2fc52d3e940c430e829135e1b811783f, entries=150, sequenceid=444, filesize=11.9 K 2023-04-09 09:54:53,247 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 026e485e431b415e1bb021c3d14de88b in 462ms, sequenceid=444, compaction requested=true 2023-04-09 09:54:53,247 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 026e485e431b415e1bb021c3d14de88b: 2023-04-09 09:54:53,247 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:54:53,247 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:54:53,247 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-09 09:54:53,247 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:54:53,247 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-09 09:54:53,248 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39612 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-09 09:54:53,248 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-09 09:54:53,248 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1912): 026e485e431b415e1bb021c3d14de88b/A is initiating minor compaction (all files) 2023-04-09 09:54:53,248 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): 026e485e431b415e1bb021c3d14de88b/B is initiating minor compaction (all files) 2023-04-09 09:54:53,248 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 026e485e431b415e1bb021c3d14de88b/A in TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:53,248 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 026e485e431b415e1bb021c3d14de88b/B in TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:53,248 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/5e607fee10024a0580f41cd46797a436, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/ea78ac69f98e4f8e84169e39aad7ba50, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/6fd0c9634496420b87fa599434913b92] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp, totalSize=38.7 K 2023-04-09 09:54:53,248 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/ed8f2fbee1784e998f9951b81f43ef2b, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/abcfc87f65874425a6dfef508f0e91a5, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/4a1492c05b564fc5a8f95c0d2bad988a] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp, totalSize=36.3 K 2023-04-09 09:54:53,249 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 5e607fee10024a0580f41cd46797a436, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=408, earliestPutTs=1681034091511 2023-04-09 09:54:53,249 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting ed8f2fbee1784e998f9951b81f43ef2b, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=408, earliestPutTs=1681034091511 2023-04-09 09:54:53,249 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting ea78ac69f98e4f8e84169e39aad7ba50, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=426, earliestPutTs=1681034092152 2023-04-09 09:54:53,249 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting abcfc87f65874425a6dfef508f0e91a5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=426, earliestPutTs=1681034092152 2023-04-09 09:54:53,249 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 6fd0c9634496420b87fa599434913b92, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=444, earliestPutTs=1681034092282 2023-04-09 09:54:53,249 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 4a1492c05b564fc5a8f95c0d2bad988a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=444, earliestPutTs=1681034092298 2023-04-09 09:54:53,253 DEBUG [Thread-1715] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:53,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:53,255 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): 026e485e431b415e1bb021c3d14de88b#B#compaction#279 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:54:53,255 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] throttle.PressureAwareThroughputController(145): 026e485e431b415e1bb021c3d14de88b#A#compaction#280 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:54:53,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:53,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:53,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:53,257 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:53,257 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:53,258 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:53,258 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:53,258 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:53,258 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:53,259 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:53,259 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:53,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:53,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:53,260 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:53,260 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:53,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:53,260 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:53,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:53,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:53,261 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:53,270 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:53,270 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:53,270 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:53,270 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:53,270 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:53,271 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:53,271 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:53,271 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:53,271 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. started... 2023-04-09 09:54:53,271 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 026e485e431b415e1bb021c3d14de88b 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-09 09:54:53,271 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=A 2023-04-09 09:54:53,271 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:53,272 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=B 2023-04-09 09:54:53,272 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:53,272 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=C 2023-04-09 09:54:53,272 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:53,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 026e485e431b415e1bb021c3d14de88b 2023-04-09 09:54:53,416 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. as already flushing 2023-04-09 09:54:53,435 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:53,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56098 deadline: 1681034153432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:53,436 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:53,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56106 deadline: 1681034153433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:53,438 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:53,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56138 deadline: 1681034153435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:53,438 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:53,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56126 deadline: 1681034153435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:53,537 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:53,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56098 deadline: 1681034153536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:53,538 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:53,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56106 deadline: 1681034153537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:53,541 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:53,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56138 deadline: 1681034153539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:53,541 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:53,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56126 deadline: 1681034153539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:53,676 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/A/b1d06e51876947fdade72cfe2316b54c as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/b1d06e51876947fdade72cfe2316b54c 2023-04-09 09:54:53,676 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/B/06759505757b44b4ae6ae5b90cfde65c as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/06759505757b44b4ae6ae5b90cfde65c 2023-04-09 09:54:53,681 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 026e485e431b415e1bb021c3d14de88b/B of 026e485e431b415e1bb021c3d14de88b into 06759505757b44b4ae6ae5b90cfde65c(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:54:53,681 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 026e485e431b415e1bb021c3d14de88b/A of 026e485e431b415e1bb021c3d14de88b into b1d06e51876947fdade72cfe2316b54c(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:54:53,681 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 026e485e431b415e1bb021c3d14de88b: 2023-04-09 09:54:53,681 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b., storeName=026e485e431b415e1bb021c3d14de88b/A, priority=13, startTime=1681034093247; duration=0sec 2023-04-09 09:54:53,681 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:54:53,681 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-09 09:54:53,681 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 026e485e431b415e1bb021c3d14de88b: 2023-04-09 09:54:53,682 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b., storeName=026e485e431b415e1bb021c3d14de88b/B, priority=13, startTime=1681034093247; duration=0sec 2023-04-09 09:54:53,682 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:54:53,682 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=461 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/A/1d874aaf37f64d0984d90bdebb60651a 2023-04-09 09:54:53,682 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-09 09:54:53,682 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1912): 026e485e431b415e1bb021c3d14de88b/C is initiating minor compaction (all files) 2023-04-09 09:54:53,683 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 026e485e431b415e1bb021c3d14de88b/C in TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:53,683 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/2052ff9828184c84aa0abba422b4fcbe, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/700fc3f06f134c4e8c192323af609ac0, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/2fc52d3e940c430e829135e1b811783f] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp, totalSize=36.3 K 2023-04-09 09:54:53,683 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 2052ff9828184c84aa0abba422b4fcbe, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=408, earliestPutTs=1681034091511 2023-04-09 09:54:53,683 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 700fc3f06f134c4e8c192323af609ac0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=426, earliestPutTs=1681034092152 2023-04-09 09:54:53,684 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 2fc52d3e940c430e829135e1b811783f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=444, earliestPutTs=1681034092298 2023-04-09 09:54:53,713 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] throttle.PressureAwareThroughputController(145): 026e485e431b415e1bb021c3d14de88b#C#compaction#283 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:54:53,722 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/C/9b25c1afba7c49f3a68633202362cd48 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/9b25c1afba7c49f3a68633202362cd48 2023-04-09 09:54:53,728 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 026e485e431b415e1bb021c3d14de88b/C of 026e485e431b415e1bb021c3d14de88b into 9b25c1afba7c49f3a68633202362cd48(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:54:53,728 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 026e485e431b415e1bb021c3d14de88b: 2023-04-09 09:54:53,728 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b., storeName=026e485e431b415e1bb021c3d14de88b/C, priority=13, startTime=1681034093247; duration=0sec 2023-04-09 09:54:53,728 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:54:53,740 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:53,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56098 deadline: 1681034153739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:53,742 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:53,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56106 deadline: 1681034153741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:53,744 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:53,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56138 deadline: 1681034153743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:53,745 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:53,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56126 deadline: 1681034153743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,044 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:54,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56098 deadline: 1681034154042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,045 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:54,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56106 deadline: 1681034154044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,048 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:54,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56126 deadline: 1681034154046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,050 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:54,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56138 deadline: 1681034154048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,102 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=461 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/B/d2f6bc62c786484db0e1e1e647427358 2023-04-09 09:54:54,512 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=461 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/C/d5e7a4014a3d45f18d311c5096db71ea 2023-04-09 09:54:54,516 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/A/1d874aaf37f64d0984d90bdebb60651a as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/1d874aaf37f64d0984d90bdebb60651a 2023-04-09 09:54:54,520 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/1d874aaf37f64d0984d90bdebb60651a, entries=150, sequenceid=461, filesize=11.9 K 2023-04-09 09:54:54,520 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/B/d2f6bc62c786484db0e1e1e647427358 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/d2f6bc62c786484db0e1e1e647427358 2023-04-09 09:54:54,523 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/d2f6bc62c786484db0e1e1e647427358, entries=150, sequenceid=461, filesize=11.9 K 2023-04-09 09:54:54,524 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/C/d5e7a4014a3d45f18d311c5096db71ea as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/d5e7a4014a3d45f18d311c5096db71ea 2023-04-09 09:54:54,527 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/d5e7a4014a3d45f18d311c5096db71ea, entries=150, sequenceid=461, filesize=11.9 K 2023-04-09 09:54:54,528 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=87.22 KB/89310 for 026e485e431b415e1bb021c3d14de88b in 1257ms, sequenceid=461, compaction requested=false 2023-04-09 09:54:54,528 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 026e485e431b415e1bb021c3d14de88b: 2023-04-09 09:54:54,528 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:54,528 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:54,528 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:54,528 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,528 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:54,528 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:54,530 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,530 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:54,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:54,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:54,530 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:54,530 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:54,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:54,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:54,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:54,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:54,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:54,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c325c2c[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:54,532 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:54,532 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:54,533 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:54,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:54,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:54,533 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:54,533 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:54,533 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:54,533 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,533 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:54,535 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:54,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:54,535 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:54,536 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:54,536 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:54,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:54,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:54,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:54,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:54,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:54,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:54,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:54,538 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:54,538 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:54,538 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:54,538 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:54,538 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:54,539 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:54,539 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:54,539 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:54,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:54,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:54,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:54,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:54,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:54,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:54,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:54,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:54,540 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:54,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:54,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:54,541 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:54,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:54,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:54,545 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,545 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:54,545 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:54,545 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:54,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:54,545 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:54,545 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:54,545 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:54,546 DEBUG [Thread-1715] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:54,546 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:54,546 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:54,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:54,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:54,546 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:54,546 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:54,546 DEBUG [Thread-1715] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:54,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:54,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:54,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 026e485e431b415e1bb021c3d14de88b 2023-04-09 09:54:54,550 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 026e485e431b415e1bb021c3d14de88b 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-09 09:54:54,551 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=A 2023-04-09 09:54:54,551 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:54,551 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=B 2023-04-09 09:54:54,551 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:54,551 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=C 2023-04-09 09:54:54,551 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:54,562 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:54,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56126 deadline: 1681034154560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,563 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:54,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56138 deadline: 1681034154560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,563 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:54,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56106 deadline: 1681034154561, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,564 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:54,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56098 deadline: 1681034154562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,646 DEBUG [Thread-1715] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:54,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:54,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:54,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:54,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:54,650 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:54,650 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:54,650 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:54,650 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:54,650 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:54,650 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:54,653 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:54,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:54,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:54,653 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:54,653 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,653 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:54,654 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:54,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:54,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:54,654 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:54,654 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:54,654 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:54,654 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:54,654 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:54,654 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:54,655 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:54,655 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:54,655 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:54,655 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. started... 2023-04-09 09:54:54,655 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. as already flushing 2023-04-09 09:54:54,655 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:54,655 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:54,655 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:54,655 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,655 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:54,655 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:54,656 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,656 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:54,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:54,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:54,657 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:54,657 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:54,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:54,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:54,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:54,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:54,658 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:54,658 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@680bb04c[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:54,658 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:54,658 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:54,659 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:54,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:54,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:54,660 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:54,660 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:54,660 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:54,660 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,660 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:54,661 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,661 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:54,661 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:54,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:54,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:54,661 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:54,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:54,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:54,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:54,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:54,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:54,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,664 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:54,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56126 deadline: 1681034154663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,664 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:54,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:54,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56138 deadline: 1681034154663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,665 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:54,665 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:54,665 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:54,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:54,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56106 deadline: 1681034154664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,665 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:54,665 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:54,665 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:54,666 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:54,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56098 deadline: 1681034154665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,668 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:54,668 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:54,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:54,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:54,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:54,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:54,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:54,668 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:54,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:54,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:54,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:54,668 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:54,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:54,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:54,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:54,670 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:54,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:54,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,672 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,672 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:54,672 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:54,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:54,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:54,672 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:54,672 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:54,672 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:54,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:54,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:54,672 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:54,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:54,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:54,672 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,672 DEBUG [Thread-1715] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:54,672 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:54,672 DEBUG [Thread-1715] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:54,672 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:54,672 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:54,772 DEBUG [Thread-1715] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:54,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:54,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:54,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:54,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:54,776 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:54,776 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:54,776 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:54,776 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:54,776 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:54,776 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:54,777 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,778 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:54,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:54,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:54,778 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,778 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:54,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:54,778 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:54,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:54,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:54,779 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:54,779 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:54,779 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:54,779 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:54,779 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:54,779 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:54,779 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:54,779 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:54,780 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:54,780 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. started... 2023-04-09 09:54:54,780 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. as already flushing 2023-04-09 09:54:54,780 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:54,780 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:54,780 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:54,780 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,780 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:54,780 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:54,781 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,781 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:54,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:54,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:54,782 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:54,782 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:54,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:54,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:54,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:54,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:54,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:54,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c092ae8[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:54,783 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:54,784 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:54,785 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:54,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:54,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:54,786 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:54,786 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:54,786 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:54,786 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,786 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:54,787 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,787 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:54,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:54,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:54,787 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:54,787 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:54,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:54,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:54,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:54,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:54,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:54,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:54,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:54,790 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:54,790 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:54,790 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:54,790 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:54,790 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:54,791 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:54,791 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:54,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:54,792 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:54,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:54,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:54,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:54,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:54,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:54,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:54,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:54,792 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:54,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:54,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:54,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:54,793 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:54,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:54,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,795 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,795 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:54,795 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:54,796 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,796 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:54,796 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:54,796 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:54,796 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:54,796 DEBUG [Thread-1715] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:54,796 DEBUG [Thread-1715] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:54,796 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:54,796 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:54,796 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:54,796 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:54,796 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:54,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:54,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:54,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:54,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:54,867 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:54,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56126 deadline: 1681034154865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,868 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:54,869 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:54,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56138 deadline: 1681034154866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56098 deadline: 1681034154867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,869 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:54,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56106 deadline: 1681034154867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,896 DEBUG [Thread-1715] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:54,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:54,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:54,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:54,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:54,899 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:54,899 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:54,899 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:54,899 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:54,900 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:54,900 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:54,902 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:54,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:54,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:54,902 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:54,903 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,903 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:54,903 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:54,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:54,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:54,903 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:54,904 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:54,904 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:54,904 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:54,904 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:54,904 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:54,904 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:54,904 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:54,904 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:54,904 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. started... 2023-04-09 09:54:54,904 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. as already flushing 2023-04-09 09:54:54,904 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:54,904 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:54,904 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:54,904 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,905 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:54,905 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:54,906 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:54,906 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:54,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:54,906 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:54,906 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:54,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:54,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:54,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:54,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:54,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:54,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61ae8a68[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:54,908 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:54,908 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:54,909 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:54,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:54,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:54,909 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:54,909 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:54,909 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:54,909 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,909 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:54,910 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,910 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:54,910 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:54,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,910 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:54,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:54,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:54,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:54,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:54,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:54,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:54,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:54,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:54,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:54,913 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:54,913 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:54,913 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:54,913 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:54,913 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:54,916 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:54,916 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:54,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:54,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:54,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:54,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:54,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:54,916 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:54,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:54,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:54,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:54,916 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:54,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:54,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:54,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:54,918 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:54,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:54,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,920 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,920 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:54,920 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:54,920 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:54,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:54,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:54,920 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:54,920 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:54,920 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:54,920 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:54,920 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:54,920 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:54,920 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:54,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:54,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:54,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:54,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:54,921 DEBUG [Thread-1715] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:54,921 DEBUG [Thread-1715] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:54,959 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=482 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/A/23b2d331ad1347d2b7d89fb7fe5f4b69 2023-04-09 09:54:54,969 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=482 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/B/597dcfacd1104df3b98a2e9c27a0cdc0 2023-04-09 09:54:54,980 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=482 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/C/8e021e2fe2834b1e93ab5608740ca248 2023-04-09 09:54:54,984 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/A/23b2d331ad1347d2b7d89fb7fe5f4b69 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/23b2d331ad1347d2b7d89fb7fe5f4b69 2023-04-09 09:54:54,987 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/23b2d331ad1347d2b7d89fb7fe5f4b69, entries=150, sequenceid=482, filesize=11.9 K 2023-04-09 09:54:54,988 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/B/597dcfacd1104df3b98a2e9c27a0cdc0 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/597dcfacd1104df3b98a2e9c27a0cdc0 2023-04-09 09:54:54,993 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/597dcfacd1104df3b98a2e9c27a0cdc0, entries=150, sequenceid=482, filesize=11.9 K 2023-04-09 09:54:54,994 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/C/8e021e2fe2834b1e93ab5608740ca248 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/8e021e2fe2834b1e93ab5608740ca248 2023-04-09 09:54:54,997 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/8e021e2fe2834b1e93ab5608740ca248, entries=150, sequenceid=482, filesize=11.9 K 2023-04-09 09:54:54,998 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 026e485e431b415e1bb021c3d14de88b in 448ms, sequenceid=482, compaction requested=true 2023-04-09 09:54:54,998 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 026e485e431b415e1bb021c3d14de88b: 2023-04-09 09:54:54,998 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:54:54,998 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-09 09:54:54,998 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-09 09:54:54,998 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:54:54,998 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:54:54,999 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-09 09:54:54,999 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1912): 026e485e431b415e1bb021c3d14de88b/A is initiating minor compaction (all files) 2023-04-09 09:54:54,999 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 026e485e431b415e1bb021c3d14de88b/A in TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:54,999 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/b1d06e51876947fdade72cfe2316b54c, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/1d874aaf37f64d0984d90bdebb60651a, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/23b2d331ad1347d2b7d89fb7fe5f4b69] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp, totalSize=36.4 K 2023-04-09 09:54:55,000 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-09 09:54:55,000 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): 026e485e431b415e1bb021c3d14de88b/B is initiating minor compaction (all files) 2023-04-09 09:54:55,000 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 026e485e431b415e1bb021c3d14de88b/B in TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:55,000 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/06759505757b44b4ae6ae5b90cfde65c, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/d2f6bc62c786484db0e1e1e647427358, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/597dcfacd1104df3b98a2e9c27a0cdc0] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp, totalSize=36.4 K 2023-04-09 09:54:55,000 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting b1d06e51876947fdade72cfe2316b54c, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=444, earliestPutTs=1681034092298 2023-04-09 09:54:55,001 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 06759505757b44b4ae6ae5b90cfde65c, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=444, earliestPutTs=1681034092298 2023-04-09 09:54:55,001 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 1d874aaf37f64d0984d90bdebb60651a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=461, earliestPutTs=1681034092800 2023-04-09 09:54:55,002 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting d2f6bc62c786484db0e1e1e647427358, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=461, earliestPutTs=1681034092800 2023-04-09 09:54:55,002 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 23b2d331ad1347d2b7d89fb7fe5f4b69, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=482, earliestPutTs=1681034093432 2023-04-09 09:54:55,002 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 597dcfacd1104df3b98a2e9c27a0cdc0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=482, earliestPutTs=1681034093432 2023-04-09 09:54:55,011 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] throttle.PressureAwareThroughputController(145): 026e485e431b415e1bb021c3d14de88b#A#compaction#289 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:54:55,011 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): 026e485e431b415e1bb021c3d14de88b#B#compaction#288 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:54:55,021 DEBUG [Thread-1715] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:55,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:55,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:55,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:55,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:55,025 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:55,026 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:55,026 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:55,026 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:55,026 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:55,026 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:55,033 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:55,033 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:55,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:55,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:55,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:55,034 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:55,034 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:55,035 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:55,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:55,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:55,035 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:55,038 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/B/f0859b7198294b4d9dfa830650db9cc4 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/f0859b7198294b4d9dfa830650db9cc4 2023-04-09 09:54:55,045 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 026e485e431b415e1bb021c3d14de88b/B of 026e485e431b415e1bb021c3d14de88b into f0859b7198294b4d9dfa830650db9cc4(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:54:55,045 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 026e485e431b415e1bb021c3d14de88b: 2023-04-09 09:54:55,045 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b., storeName=026e485e431b415e1bb021c3d14de88b/B, priority=13, startTime=1681034094998; duration=0sec 2023-04-09 09:54:55,045 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:54:55,045 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-09 09:54:55,046 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-09 09:54:55,046 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:55,047 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): 026e485e431b415e1bb021c3d14de88b/C is initiating minor compaction (all files) 2023-04-09 09:54:55,047 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:55,047 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:55,047 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 026e485e431b415e1bb021c3d14de88b/C in TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:55,047 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:55,047 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/9b25c1afba7c49f3a68633202362cd48, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/d5e7a4014a3d45f18d311c5096db71ea, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/8e021e2fe2834b1e93ab5608740ca248] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp, totalSize=36.4 K 2023-04-09 09:54:55,047 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:55,047 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:55,047 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:55,047 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. started... 2023-04-09 09:54:55,047 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 9b25c1afba7c49f3a68633202362cd48, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=444, earliestPutTs=1681034092298 2023-04-09 09:54:55,047 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 026e485e431b415e1bb021c3d14de88b 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-09 09:54:55,048 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting d5e7a4014a3d45f18d311c5096db71ea, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=461, earliestPutTs=1681034092800 2023-04-09 09:54:55,048 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=A 2023-04-09 09:54:55,048 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:55,048 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=B 2023-04-09 09:54:55,048 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 8e021e2fe2834b1e93ab5608740ca248, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=482, earliestPutTs=1681034093432 2023-04-09 09:54:55,048 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:55,049 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=C 2023-04-09 09:54:55,049 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:55,058 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): 026e485e431b415e1bb021c3d14de88b#C#compaction#291 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:54:55,108 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=497 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/A/1c0ac7d6c32a4087a6fa09be413d302d 2023-04-09 09:54:55,153 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=497 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/B/d36a0008f5c0482c9da4f8984fd87caa 2023-04-09 09:54:55,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 026e485e431b415e1bb021c3d14de88b 2023-04-09 09:54:55,171 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. as already flushing 2023-04-09 09:54:55,207 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:55,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56106 deadline: 1681034155205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:55,209 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:55,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56138 deadline: 1681034155206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:55,209 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:55,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56098 deadline: 1681034155207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:55,209 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:55,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56126 deadline: 1681034155207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:55,309 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:55,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56106 deadline: 1681034155308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:55,311 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:55,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56098 deadline: 1681034155310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:55,311 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:55,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56138 deadline: 1681034155310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:55,311 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:55,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56126 deadline: 1681034155310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:55,422 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/A/2cd94d187dac4bdabcfe41f807dff3ff as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/2cd94d187dac4bdabcfe41f807dff3ff 2023-04-09 09:54:55,426 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 026e485e431b415e1bb021c3d14de88b/A of 026e485e431b415e1bb021c3d14de88b into 2cd94d187dac4bdabcfe41f807dff3ff(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:54:55,426 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 026e485e431b415e1bb021c3d14de88b: 2023-04-09 09:54:55,426 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b., storeName=026e485e431b415e1bb021c3d14de88b/A, priority=13, startTime=1681034094998; duration=0sec 2023-04-09 09:54:55,426 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:54:55,511 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:55,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56106 deadline: 1681034155510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:55,514 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:55,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 287 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56126 deadline: 1681034155512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:55,515 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:55,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56098 deadline: 1681034155513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:55,515 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:55,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56138 deadline: 1681034155513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:55,525 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/C/bdee1af652c040af8461c11efba78d10 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/bdee1af652c040af8461c11efba78d10 2023-04-09 09:54:55,534 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 026e485e431b415e1bb021c3d14de88b/C of 026e485e431b415e1bb021c3d14de88b into bdee1af652c040af8461c11efba78d10(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:54:55,534 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 026e485e431b415e1bb021c3d14de88b: 2023-04-09 09:54:55,534 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b., storeName=026e485e431b415e1bb021c3d14de88b/C, priority=13, startTime=1681034094998; duration=0sec 2023-04-09 09:54:55,534 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:54:55,570 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=497 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/C/1bb06a59bcfc47e0bdb4b891ddcfbbfb 2023-04-09 09:54:55,574 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/A/1c0ac7d6c32a4087a6fa09be413d302d as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/1c0ac7d6c32a4087a6fa09be413d302d 2023-04-09 09:54:55,578 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/1c0ac7d6c32a4087a6fa09be413d302d, entries=100, sequenceid=497, filesize=9.5 K 2023-04-09 09:54:55,579 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/B/d36a0008f5c0482c9da4f8984fd87caa as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/d36a0008f5c0482c9da4f8984fd87caa 2023-04-09 09:54:55,583 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/d36a0008f5c0482c9da4f8984fd87caa, entries=100, sequenceid=497, filesize=9.5 K 2023-04-09 09:54:55,584 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/C/1bb06a59bcfc47e0bdb4b891ddcfbbfb as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/1bb06a59bcfc47e0bdb4b891ddcfbbfb 2023-04-09 09:54:55,587 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/1bb06a59bcfc47e0bdb4b891ddcfbbfb, entries=100, sequenceid=497, filesize=9.5 K 2023-04-09 09:54:55,588 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for 026e485e431b415e1bb021c3d14de88b in 541ms, sequenceid=497, compaction requested=false 2023-04-09 09:54:55,588 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 026e485e431b415e1bb021c3d14de88b: 2023-04-09 09:54:55,588 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:55,588 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:55,588 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:55,588 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:55,588 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:55,588 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:55,590 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:55,590 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:55,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:55,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:55,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:55,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:55,601 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:55,601 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:55,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:55,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:55,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:55,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:55,602 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:55,602 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ff3694d[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:55,602 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:55,602 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:55,604 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:55,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:55,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:55,604 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:55,604 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:55,604 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:55,604 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:55,604 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:55,606 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:55,606 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:55,606 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:55,606 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:55,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:55,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:55,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:55,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:55,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:55,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:55,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:55,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:55,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:55,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:55,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:55,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:55,609 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:55,609 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:55,609 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:55,609 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:55,609 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:55,610 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:55,610 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:55,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:55,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:55,610 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:55,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:55,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:55,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:55,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:55,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:55,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:55,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:55,611 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:55,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:55,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:55,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:55,612 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:55,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:55,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:55,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:55,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:55,615 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:55,615 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:55,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:55,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:55,615 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:55,615 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:55,615 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:55,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:55,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:55,615 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:55,615 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:55,615 DEBUG [Thread-1715] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:55,615 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:55,616 DEBUG [Thread-1715] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:55,616 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:55,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:55,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:55,616 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:55,616 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:55,716 DEBUG [Thread-1715] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:55,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:55,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:55,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:55,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:55,720 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:55,720 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:55,720 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:55,720 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:55,720 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:55,720 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:55,722 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:55,722 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:55,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:55,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:55,722 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:55,722 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:55,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:55,722 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:55,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:55,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:55,723 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:55,723 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:55,723 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:55,723 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:55,723 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:55,723 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:55,724 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:55,724 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:55,724 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:55,724 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. started... 2023-04-09 09:54:55,724 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 026e485e431b415e1bb021c3d14de88b 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-09 09:54:55,724 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=A 2023-04-09 09:54:55,724 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:55,725 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=B 2023-04-09 09:54:55,725 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:55,725 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=C 2023-04-09 09:54:55,725 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:55,742 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=518 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/A/10e74cf2ce0d4f2ea9976a89950bd8b9 2023-04-09 09:54:55,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 026e485e431b415e1bb021c3d14de88b 2023-04-09 09:54:55,815 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. as already flushing 2023-04-09 09:54:55,830 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:55,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56106 deadline: 1681034155828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:55,831 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:55,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56138 deadline: 1681034155828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:55,831 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:55,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56098 deadline: 1681034155828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:55,831 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:55,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56126 deadline: 1681034155829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:55,933 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:55,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56106 deadline: 1681034155932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:55,933 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:55,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56138 deadline: 1681034155932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:55,934 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:55,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 294 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56126 deadline: 1681034155932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:55,934 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:55,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56098 deadline: 1681034155932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:56,136 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:56,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56138 deadline: 1681034156135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:56,137 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:56,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56106 deadline: 1681034156136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:56,137 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:56,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 296 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56126 deadline: 1681034156136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:56,137 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:56,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56098 deadline: 1681034156136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:56,162 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=518 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/B/5ced19337b04406cace2075d31611bbf 2023-04-09 09:54:56,172 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=518 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/C/2ae74df5615a47bb894c62dcdc3184a5 2023-04-09 09:54:56,175 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/A/10e74cf2ce0d4f2ea9976a89950bd8b9 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/10e74cf2ce0d4f2ea9976a89950bd8b9 2023-04-09 09:54:56,183 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/10e74cf2ce0d4f2ea9976a89950bd8b9, entries=150, sequenceid=518, filesize=11.9 K 2023-04-09 09:54:56,184 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/B/5ced19337b04406cace2075d31611bbf as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/5ced19337b04406cace2075d31611bbf 2023-04-09 09:54:56,190 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/5ced19337b04406cace2075d31611bbf, entries=150, sequenceid=518, filesize=11.9 K 2023-04-09 09:54:56,190 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/C/2ae74df5615a47bb894c62dcdc3184a5 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/2ae74df5615a47bb894c62dcdc3184a5 2023-04-09 09:54:56,197 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/2ae74df5615a47bb894c62dcdc3184a5, entries=150, sequenceid=518, filesize=11.9 K 2023-04-09 09:54:56,197 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=80.51 KB/82440 for 026e485e431b415e1bb021c3d14de88b in 473ms, sequenceid=518, compaction requested=true 2023-04-09 09:54:56,197 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 026e485e431b415e1bb021c3d14de88b: 2023-04-09 09:54:56,197 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:56,197 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:56,198 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:56,198 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:56,198 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:56,198 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:56,199 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:56,199 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:56,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:56,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:56,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:56,200 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:56,200 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:56,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:56,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:56,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:56,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:56,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:56,206 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:56,206 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a4f2d2f[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:56,206 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:56,206 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:56,207 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:56,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:56,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:56,207 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:56,207 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:56,207 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:56,207 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:56,208 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:56,209 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:56,209 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:56,209 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:56,209 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:56,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:56,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:56,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:56,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:56,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:56,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:56,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:56,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:56,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:56,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:56,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:56,212 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:56,212 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:56,212 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:56,212 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:56,212 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:56,212 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:56,216 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:56,216 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:56,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:56,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:56,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:56,216 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:56,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:56,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:56,216 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:56,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:56,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:56,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:56,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:56,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:56,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:56,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:56,218 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:56,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:56,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:56,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:56,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:56,220 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:56,220 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:56,220 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:56,220 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:56,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:56,220 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:56,220 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:56,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:56,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:56,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:56,220 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:56,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:56,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:56,220 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:56,221 DEBUG [Thread-1715] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:56,221 DEBUG [Thread-1715] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:56,221 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:56,221 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:56,221 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:56,321 DEBUG [Thread-1715] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:56,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:56,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:56,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:56,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:56,325 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:56,325 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:56,325 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:56,325 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:56,325 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:56,325 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:56,327 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:56,327 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:56,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:56,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:56,327 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:56,327 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:56,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:56,327 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:56,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:56,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:56,328 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:56,328 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:56,328 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:56,328 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:56,328 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:56,328 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:56,329 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:56,329 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:56,329 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:56,329 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. started... 2023-04-09 09:54:56,329 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 026e485e431b415e1bb021c3d14de88b 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-09 09:54:56,329 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=A 2023-04-09 09:54:56,329 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:56,329 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=B 2023-04-09 09:54:56,329 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:56,329 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=C 2023-04-09 09:54:56,329 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:56,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 026e485e431b415e1bb021c3d14de88b 2023-04-09 09:54:56,441 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. as already flushing 2023-04-09 09:54:56,456 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:56,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56106 deadline: 1681034156452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:56,456 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:56,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56126 deadline: 1681034156452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:56,456 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:56,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56138 deadline: 1681034156452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:56,458 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:56,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56098 deadline: 1681034156455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:56,559 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:56,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56106 deadline: 1681034156557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:56,560 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:56,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56138 deadline: 1681034156557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:56,560 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:56,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 304 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56126 deadline: 1681034156557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:56,561 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:56,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56098 deadline: 1681034156559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:56,736 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=533 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/A/fdccd7f13a6840a1afc0e443aac49aae 2023-04-09 09:54:56,748 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=533 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/B/44943f53ea9c440fad45f9c9df9ea211 2023-04-09 09:54:56,762 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:56,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56106 deadline: 1681034156761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:56,763 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:56,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56098 deadline: 1681034156762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:56,763 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:56,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 306 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56126 deadline: 1681034156762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:56,763 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:56,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56138 deadline: 1681034156762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,065 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:57,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56138 deadline: 1681034157064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,066 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:57,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56106 deadline: 1681034157065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,067 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:57,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 308 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56126 deadline: 1681034157065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,068 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:57,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56098 deadline: 1681034157066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,161 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=533 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/C/299a0beab0a7495da18fd0be015886bc 2023-04-09 09:54:57,165 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/A/fdccd7f13a6840a1afc0e443aac49aae as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/fdccd7f13a6840a1afc0e443aac49aae 2023-04-09 09:54:57,169 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/fdccd7f13a6840a1afc0e443aac49aae, entries=150, sequenceid=533, filesize=11.9 K 2023-04-09 09:54:57,169 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/B/44943f53ea9c440fad45f9c9df9ea211 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/44943f53ea9c440fad45f9c9df9ea211 2023-04-09 09:54:57,173 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/44943f53ea9c440fad45f9c9df9ea211, entries=150, sequenceid=533, filesize=11.9 K 2023-04-09 09:54:57,173 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/C/299a0beab0a7495da18fd0be015886bc as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/299a0beab0a7495da18fd0be015886bc 2023-04-09 09:54:57,177 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/299a0beab0a7495da18fd0be015886bc, entries=150, sequenceid=533, filesize=11.9 K 2023-04-09 09:54:57,178 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 026e485e431b415e1bb021c3d14de88b in 849ms, sequenceid=533, compaction requested=true 2023-04-09 09:54:57,178 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 026e485e431b415e1bb021c3d14de88b: 2023-04-09 09:54:57,178 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:57,178 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:57,178 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:57,178 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,178 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:57,178 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:57,180 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:57,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:57,180 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:57,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:57,180 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:57,180 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:57,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:57,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:57,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:57,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:57,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@fc402e4[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:57,181 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:57,181 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:57,184 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:57,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:57,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:57,184 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:57,184 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:57,184 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:57,184 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,184 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:57,186 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:57,186 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:57,186 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,186 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:57,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:57,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:57,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:57,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:57,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:57,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:57,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:57,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:57,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:57,189 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:57,189 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:57,189 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:57,189 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:57,189 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:57,192 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:57,192 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:57,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:57,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:57,192 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:57,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:57,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:57,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:57,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:57,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:57,192 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:57,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:57,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:57,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:57,194 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:57,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:57,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:57,197 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,197 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:57,197 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:57,197 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:57,197 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:57,197 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:57,197 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:57,197 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:57,197 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:57,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:57,197 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:57,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:57,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:57,198 DEBUG [Thread-1715] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:57,198 DEBUG [Thread-1715] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:57,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:57,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:57,198 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:57,298 DEBUG [Thread-1715] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:57,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:57,300 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:57,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:57,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:57,301 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:57,301 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:57,301 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:57,301 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:57,302 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:57,302 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:57,304 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,304 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:57,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:57,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:57,304 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,304 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:57,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:57,304 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:57,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:57,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:57,305 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:57,305 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:57,305 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:57,305 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:57,305 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:57,305 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:57,305 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:57,305 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:57,305 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:57,305 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. started... 2023-04-09 09:54:57,305 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 026e485e431b415e1bb021c3d14de88b 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-09 09:54:57,306 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=A 2023-04-09 09:54:57,306 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:57,306 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=B 2023-04-09 09:54:57,306 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:57,306 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=C 2023-04-09 09:54:57,306 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:57,312 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=550 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/A/20751b05a1d14e969751dc0161360aa2 2023-04-09 09:54:57,322 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=550 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/B/1c28feb4f1f04e1a8acb481d8596bc49 2023-04-09 09:54:57,331 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=550 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/C/08525928d962485fb1f7c4ab5ee2b0ef 2023-04-09 09:54:57,335 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/A/20751b05a1d14e969751dc0161360aa2 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/20751b05a1d14e969751dc0161360aa2 2023-04-09 09:54:57,338 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/20751b05a1d14e969751dc0161360aa2, entries=150, sequenceid=550, filesize=11.9 K 2023-04-09 09:54:57,339 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/B/1c28feb4f1f04e1a8acb481d8596bc49 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/1c28feb4f1f04e1a8acb481d8596bc49 2023-04-09 09:54:57,343 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/1c28feb4f1f04e1a8acb481d8596bc49, entries=150, sequenceid=550, filesize=11.9 K 2023-04-09 09:54:57,344 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/C/08525928d962485fb1f7c4ab5ee2b0ef as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/08525928d962485fb1f7c4ab5ee2b0ef 2023-04-09 09:54:57,348 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/08525928d962485fb1f7c4ab5ee2b0ef, entries=150, sequenceid=550, filesize=11.9 K 2023-04-09 09:54:57,349 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=0 B/0 for 026e485e431b415e1bb021c3d14de88b in 44ms, sequenceid=550, compaction requested=true 2023-04-09 09:54:57,349 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 026e485e431b415e1bb021c3d14de88b: 2023-04-09 09:54:57,349 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:57,349 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:57,349 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:57,349 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,356 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:57,356 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:57,361 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,361 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:57,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:57,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:57,362 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:57,362 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:57,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:57,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:57,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:57,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:57,364 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:57,364 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a2c34be[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:57,364 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:57,364 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:57,366 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:57,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:57,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:57,366 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:57,366 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:57,366 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:57,366 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,366 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:57,367 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:57,367 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:57,367 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:57,368 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:57,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:57,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:57,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:57,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:57,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:57,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:57,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:57,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:57,371 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:57,371 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:57,372 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:57,372 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:57,372 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:57,374 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:57,374 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:57,374 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:57,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:57,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:57,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:57,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:57,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:57,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:57,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:57,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:57,375 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:57,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:57,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:57,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:57,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,377 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:57,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:57,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,379 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,379 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:57,379 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:57,379 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:57,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:57,379 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:57,380 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,380 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:57,380 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:57,380 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:57,379 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:57,380 DEBUG [Thread-1715] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:57,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:57,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:57,380 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:57,380 DEBUG [Thread-1715] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:57,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:57,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:57,381 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:57,480 DEBUG [Thread-1715] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:57,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:57,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:57,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:57,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:57,484 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:57,484 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:57,484 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:57,484 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:57,484 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:57,484 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:57,486 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:57,486 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:57,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:57,486 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:57,486 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:57,486 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:57,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:57,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:57,487 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:57,487 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:57,487 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:57,487 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:57,487 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:57,487 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:57,487 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:57,488 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:57,488 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:57,488 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. started... 2023-04-09 09:54:57,488 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 026e485e431b415e1bb021c3d14de88b: 2023-04-09 09:54:57,488 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:57,488 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:57,488 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:57,488 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,488 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:57,488 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:57,489 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,489 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:57,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:57,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:57,490 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:57,490 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:57,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:57,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:57,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:57,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:57,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:57,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ee18fb7[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:57,491 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:57,491 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:57,493 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:57,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:57,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:57,493 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:57,493 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,493 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:57,493 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:57,493 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:57,494 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,494 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:57,495 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:57,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:57,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:57,495 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:57,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:57,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:57,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:57,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:57,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:57,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:57,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:57,497 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:57,497 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:57,497 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:57,497 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:57,497 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:57,499 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:57,499 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:57,499 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:57,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:57,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:57,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:57,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:57,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:57,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:57,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:57,499 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:57,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:57,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:57,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:57,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:57,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,501 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:57,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:57,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,503 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,503 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:57,503 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:57,503 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:57,503 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:57,503 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,503 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:57,503 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:57,503 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:57,503 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:57,504 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:57,504 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:57,504 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:57,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:57,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:57,504 DEBUG [Thread-1715] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:57,504 DEBUG [Thread-1715] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:57,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:57,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:57,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 026e485e431b415e1bb021c3d14de88b 2023-04-09 09:54:57,575 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 026e485e431b415e1bb021c3d14de88b 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-09 09:54:57,575 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=A 2023-04-09 09:54:57,575 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:57,575 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=B 2023-04-09 09:54:57,575 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:57,575 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=C 2023-04-09 09:54:57,575 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:57,596 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=562 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/A/a4dfdfac409f49a1aa95b2beb45b6222 2023-04-09 09:54:57,602 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:57,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56106 deadline: 1681034157595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,602 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:57,602 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681034097591, max timestamp=1681034097599Num uniques 859; 2023-04-09 09:54:57,602 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681034097591, max timestamp=1681034097599Num uniques 859; 2023-04-09 09:54:57,603 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-09 09:54:57,602 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681034097591, max timestamp=1681034097599Num uniques 859; 2023-04-09 09:54:57,603 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-09 09:54:57,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56098 deadline: 1681034157596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,603 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-09 09:54:57,604 DEBUG [Thread-1715] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:57,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:57,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:57,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:57,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:57,616 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:57,616 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:57,616 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:57,616 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:57,616 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:57,616 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:57,619 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:57,619 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:57,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:57,619 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,619 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:57,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:57,620 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:57,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:57,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:57,620 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:57,632 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:57,632 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:57,632 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:57,632 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:57,632 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:57,632 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:57,632 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:57,632 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:57,632 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. started... 2023-04-09 09:54:57,632 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. as already flushing 2023-04-09 09:54:57,632 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:57,633 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:57,633 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:57,633 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,633 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:57,633 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:57,634 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,634 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:57,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:57,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:57,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:57,635 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:57,635 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:57,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:57,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:57,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:57,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:57,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d64f111[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:57,636 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:57,636 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:57,638 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:57,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:57,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:57,638 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:57,638 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:57,638 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:57,638 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,638 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:57,639 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,639 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:57,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:57,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:57,639 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:57,639 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:57,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:57,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:57,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:57,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:57,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:57,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:57,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:57,641 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:57,641 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:57,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 324 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56126 deadline: 1681034157637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,642 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:57,642 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:57,642 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:57,642 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:57,642 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:57,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56138 deadline: 1681034157637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,644 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:57,644 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:57,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:57,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:57,644 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:57,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:57,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:57,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:57,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:57,645 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:57,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:57,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:57,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:57,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:57,646 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:57,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:57,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:57,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,648 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,648 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:57,648 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:57,648 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,648 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:57,648 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:57,648 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:57,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:57,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:57,648 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:57,648 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:57,649 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:57,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:57,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:57,649 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:57,649 DEBUG [Thread-1715] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:57,649 DEBUG [Thread-1715] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:57,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:57,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:57,706 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:57,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56106 deadline: 1681034157703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,706 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:57,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56098 deadline: 1681034157704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,745 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:57,745 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:57,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 326 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56126 deadline: 1681034157742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56138 deadline: 1681034157743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,749 DEBUG [Thread-1715] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:57,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:57,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:57,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:57,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:57,752 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:57,752 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:57,752 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:57,752 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:57,753 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:57,753 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:57,754 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,754 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:57,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:57,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:57,755 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,755 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:57,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:57,755 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:57,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:57,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:57,756 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:57,756 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:57,756 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:57,756 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:57,756 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:57,756 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:57,756 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:57,756 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:57,756 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:57,757 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. started... 2023-04-09 09:54:57,757 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. as already flushing 2023-04-09 09:54:57,757 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:57,757 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:57,757 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:57,757 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,757 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:57,757 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:57,762 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:57,762 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:57,762 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:57,763 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:57,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:57,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:57,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:57,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:57,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:57,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:57,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6298197b[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:57,765 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:57,765 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:57,767 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:57,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:57,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:57,767 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:57,767 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:57,767 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:57,768 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,768 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:57,772 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,772 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:57,772 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:57,772 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:57,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:57,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:57,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:57,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:57,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:57,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:57,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:57,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:57,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:57,776 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:57,776 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:57,776 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:57,776 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:57,776 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:57,778 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:57,778 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:57,778 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:57,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:57,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:57,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:57,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:57,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:57,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:57,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:57,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:57,779 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:57,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:57,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:57,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:57,781 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:57,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:57,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,797 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:57,797 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:57,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:57,797 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:57,797 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:57,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:57,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:57,797 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:57,797 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:57,797 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,797 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:57,798 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:57,798 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:57,798 DEBUG [Thread-1715] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:57,798 DEBUG [Thread-1715] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:57,798 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:57,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:57,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:57,898 DEBUG [Thread-1715] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:57,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:57,900 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:57,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:57,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:57,901 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:57,901 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:57,901 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:57,901 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:57,901 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:57,901 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:57,903 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,903 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:57,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:57,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:57,904 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,904 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:57,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:57,904 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:57,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:57,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:57,905 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:57,905 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:57,905 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:57,905 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:57,905 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:57,905 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:57,905 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:57,905 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:57,905 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. started... 2023-04-09 09:54:57,905 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. as already flushing 2023-04-09 09:54:57,905 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:57,905 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:57,905 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:57,905 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,905 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:57,905 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:57,907 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:57,907 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:57,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:57,907 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:57,907 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:57,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:57,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:57,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:57,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:57,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:57,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@196a4852[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:57,909 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:57,909 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:57,910 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:57,910 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:57,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56106 deadline: 1681034157907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:57,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:57,910 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:57,910 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:57,910 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:57,910 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,910 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:57,910 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:57,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56098 deadline: 1681034157907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,911 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,911 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:57,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:57,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:57,911 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:57,911 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:57,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:57,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:57,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:57,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:57,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:57,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:57,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:57,914 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:57,914 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:57,914 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:57,914 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:57,914 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:57,916 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:57,916 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:57,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:57,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:57,916 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:57,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:57,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:57,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:57,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:57,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:57,917 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:57,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:57,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:57,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:57,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:57,918 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:57,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:57,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,920 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:57,920 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:57,920 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:57,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:57,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:57,920 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:57,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:57,920 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:57,920 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:57,920 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,920 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:57,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:57,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:57,920 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:57,920 DEBUG [Thread-1715] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:57,920 DEBUG [Thread-1715] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:57,920 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:57,921 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:57,949 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:57,949 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:57,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 328 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56126 deadline: 1681034157946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:57,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56138 deadline: 1681034157946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,021 DEBUG [Thread-1715] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:58,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:58,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:58,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:58,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:58,024 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:58,024 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:58,024 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:58,024 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:58,025 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:58,025 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:58,026 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=562 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/B/b748fb1d35074d4fa307099a595ea43c 2023-04-09 09:54:58,026 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:58,026 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:58,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:58,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:58,027 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,027 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:58,027 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:58,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:58,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:58,027 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:58,027 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:58,027 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:58,028 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:58,028 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:58,028 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:58,028 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:58,028 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:58,028 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:58,028 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. started... 2023-04-09 09:54:58,028 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. as already flushing 2023-04-09 09:54:58,028 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:58,028 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:58,028 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:58,028 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,028 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:58,029 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:58,030 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:58,030 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:58,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:58,031 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:58,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:58,031 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:58,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:58,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:58,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:58,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:58,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b94f0ff[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:58,032 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:58,032 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:58,035 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,035 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:58,035 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:58,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:58,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:58,035 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:58,035 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:58,035 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:58,037 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,037 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:58,038 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:58,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:58,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:58,038 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:58,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:58,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:58,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:58,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:58,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:58,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:58,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:58,041 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:58,042 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:58,042 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:58,042 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:58,042 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:58,043 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:58,043 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:58,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:58,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:58,043 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:58,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:58,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:58,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:58,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:58,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:58,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:58,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:58,045 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:58,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:58,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:58,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,046 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:58,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:58,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,050 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,050 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:58,050 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:58,050 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:58,050 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:58,050 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:58,050 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:58,050 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:58,050 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:58,050 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,050 DEBUG [Thread-1715] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:58,050 DEBUG [Thread-1715] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:58,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:58,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:58,050 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:58,051 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:58,051 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:58,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:58,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:58,150 DEBUG [Thread-1715] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:58,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:58,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:58,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:58,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:58,153 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:58,153 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:58,154 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:58,154 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:58,154 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:58,154 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:58,156 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:58,156 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:58,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:58,156 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,156 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:58,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:58,157 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:58,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:58,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:58,157 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:58,157 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:58,157 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:58,157 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:58,157 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:58,157 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:58,158 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:58,158 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:58,158 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:58,158 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. started... 2023-04-09 09:54:58,158 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. as already flushing 2023-04-09 09:54:58,158 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:58,158 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:58,158 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:58,158 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,158 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:58,158 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:58,159 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,159 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:58,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:58,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:58,160 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:58,160 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:58,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:58,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:58,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:58,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:58,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:58,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11927ae5[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:58,161 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:58,161 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:58,162 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:58,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:58,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:58,162 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:58,162 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:58,162 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:58,162 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,162 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:58,164 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,164 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:58,164 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:58,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:58,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:58,164 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:58,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:58,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:58,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:58,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:58,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:58,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:58,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:58,166 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:58,166 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:58,166 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:58,166 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:58,166 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:58,171 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:58,171 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:58,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:58,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:58,171 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:58,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:58,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:58,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:58,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:58,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:58,171 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:58,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:58,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:58,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:58,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:58,172 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:58,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:58,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,174 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:58,174 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:58,174 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:58,174 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:58,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:58,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:58,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:58,175 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:54:58,174 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:58,175 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:58,175 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,175 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:58,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:58,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:58,175 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:58,175 DEBUG [Thread-1715] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:58,175 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:58,175 DEBUG [Thread-1715] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:58,215 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:58,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56098 deadline: 1681034158212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,215 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:58,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56106 deadline: 1681034158213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,253 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:58,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56138 deadline: 1681034158251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,253 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:58,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 330 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56126 deadline: 1681034158252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,275 DEBUG [Thread-1715] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:58,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:58,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:58,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:58,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:58,278 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:58,279 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:58,279 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:58,279 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:58,279 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:58,279 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:58,280 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:58,280 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:58,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:58,281 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,281 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:58,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:58,281 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:58,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:58,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:58,281 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:58,282 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:58,282 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:58,282 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:58,282 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:58,282 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:58,282 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:58,282 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:58,282 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:58,282 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. started... 2023-04-09 09:54:58,282 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. as already flushing 2023-04-09 09:54:58,282 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:58,282 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:58,282 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:58,282 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,282 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:58,282 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:58,285 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,285 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:58,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:58,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:58,285 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:58,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:58,285 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:58,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:58,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:58,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:58,286 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:58,286 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33e3ea6d[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:58,286 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:58,287 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:58,288 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:58,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:58,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:58,288 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:58,288 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:58,288 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:58,288 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,288 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:58,289 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,289 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:58,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:58,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:58,289 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:58,290 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:58,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:58,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:58,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:58,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:58,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:58,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:58,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:58,292 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:58,292 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:58,292 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:58,292 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:58,292 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:58,294 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:58,294 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:58,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:58,294 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:58,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:58,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:58,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:58,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:58,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:58,294 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:58,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:58,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:58,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:58,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:58,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:58,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,296 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:58,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:58,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,299 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,299 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:58,299 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:58,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:58,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:58,299 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:58,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:58,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:58,299 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:58,299 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:58,299 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:58,299 DEBUG [Thread-1715] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:58,299 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:58,300 DEBUG [Thread-1715] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:58,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:58,300 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:58,300 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:58,400 DEBUG [Thread-1715] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:58,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:58,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:58,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:58,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:58,403 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:58,403 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:58,403 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:58,403 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:58,404 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:58,404 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:58,405 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:58,405 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:58,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:58,405 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:58,406 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:58,406 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:58,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:58,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:58,406 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:58,406 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:58,406 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:58,406 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:58,407 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:58,407 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:58,407 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:58,407 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:58,407 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:58,407 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. started... 2023-04-09 09:54:58,407 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. as already flushing 2023-04-09 09:54:58,407 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:58,407 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:58,407 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:58,407 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,407 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:58,407 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:58,409 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,409 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:58,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:58,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:58,409 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:58,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:58,409 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:58,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:58,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:58,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:58,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:58,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78001c9b[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:58,411 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:58,411 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:58,412 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:58,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:58,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:58,412 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:58,412 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:58,412 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,412 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:58,412 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:58,415 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,415 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:58,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:58,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:58,415 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:58,415 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:58,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:58,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:58,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:58,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:58,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:58,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:58,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:58,420 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:58,420 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:58,420 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:58,421 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:58,421 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:58,422 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:58,422 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:58,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:58,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:58,422 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:58,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:58,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:58,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:58,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:58,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:58,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:58,423 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:58,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:58,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:58,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:58,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,424 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:58,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:58,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,426 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:58,426 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:58,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:58,426 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:58,426 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:58,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:58,426 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:58,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:58,426 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:58,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:58,427 DEBUG [Thread-1715] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:58,427 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:58,427 DEBUG [Thread-1715] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:58,427 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,427 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:58,427 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:58,427 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:58,437 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=562 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/C/bf33f8604b2b42cbb842fc65f57d08b6 2023-04-09 09:54:58,444 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/A/a4dfdfac409f49a1aa95b2beb45b6222 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/a4dfdfac409f49a1aa95b2beb45b6222 2023-04-09 09:54:58,448 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/a4dfdfac409f49a1aa95b2beb45b6222, entries=300, sequenceid=562, filesize=19.0 K 2023-04-09 09:54:58,448 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/B/b748fb1d35074d4fa307099a595ea43c as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/b748fb1d35074d4fa307099a595ea43c 2023-04-09 09:54:58,452 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/b748fb1d35074d4fa307099a595ea43c, entries=150, sequenceid=562, filesize=11.9 K 2023-04-09 09:54:58,453 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/C/bf33f8604b2b42cbb842fc65f57d08b6 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/bf33f8604b2b42cbb842fc65f57d08b6 2023-04-09 09:54:58,458 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/bf33f8604b2b42cbb842fc65f57d08b6, entries=150, sequenceid=562, filesize=11.9 K 2023-04-09 09:54:58,459 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 026e485e431b415e1bb021c3d14de88b in 884ms, sequenceid=562, compaction requested=true 2023-04-09 09:54:58,459 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 026e485e431b415e1bb021c3d14de88b: 2023-04-09 09:54:58,459 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:54:58,459 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:54:58,459 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-09 09:54:58,459 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-09 09:54:58,459 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:54:58,461 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 78810 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-09 09:54:58,461 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1912): 026e485e431b415e1bb021c3d14de88b/A is initiating minor compaction (all files) 2023-04-09 09:54:58,461 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71490 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-09 09:54:58,461 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 026e485e431b415e1bb021c3d14de88b/A in TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:58,461 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): 026e485e431b415e1bb021c3d14de88b/B is initiating minor compaction (all files) 2023-04-09 09:54:58,461 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 026e485e431b415e1bb021c3d14de88b/B in TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:58,461 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/2cd94d187dac4bdabcfe41f807dff3ff, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/1c0ac7d6c32a4087a6fa09be413d302d, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/10e74cf2ce0d4f2ea9976a89950bd8b9, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/fdccd7f13a6840a1afc0e443aac49aae, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/20751b05a1d14e969751dc0161360aa2, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/a4dfdfac409f49a1aa95b2beb45b6222] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp, totalSize=77.0 K 2023-04-09 09:54:58,461 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/f0859b7198294b4d9dfa830650db9cc4, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/d36a0008f5c0482c9da4f8984fd87caa, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/5ced19337b04406cace2075d31611bbf, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/44943f53ea9c440fad45f9c9df9ea211, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/1c28feb4f1f04e1a8acb481d8596bc49, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/b748fb1d35074d4fa307099a595ea43c] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp, totalSize=69.8 K 2023-04-09 09:54:58,462 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 2cd94d187dac4bdabcfe41f807dff3ff, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=482, earliestPutTs=1681034093432 2023-04-09 09:54:58,462 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting f0859b7198294b4d9dfa830650db9cc4, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=482, earliestPutTs=1681034093432 2023-04-09 09:54:58,462 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 1c0ac7d6c32a4087a6fa09be413d302d, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=497, earliestPutTs=1681034094559 2023-04-09 09:54:58,462 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 10e74cf2ce0d4f2ea9976a89950bd8b9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=518, earliestPutTs=1681034095179 2023-04-09 09:54:58,462 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting d36a0008f5c0482c9da4f8984fd87caa, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=497, earliestPutTs=1681034094559 2023-04-09 09:54:58,463 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting fdccd7f13a6840a1afc0e443aac49aae, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=533, earliestPutTs=1681034095827 2023-04-09 09:54:58,463 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 5ced19337b04406cace2075d31611bbf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=518, earliestPutTs=1681034095179 2023-04-09 09:54:58,463 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 20751b05a1d14e969751dc0161360aa2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=550, earliestPutTs=1681034096447 2023-04-09 09:54:58,463 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 44943f53ea9c440fad45f9c9df9ea211, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=533, earliestPutTs=1681034095827 2023-04-09 09:54:58,463 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting a4dfdfac409f49a1aa95b2beb45b6222, keycount=300, bloomtype=ROW, size=19.0 K, encoding=NONE, compression=NONE, seqNum=562, earliestPutTs=1681034097571 2023-04-09 09:54:58,463 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 1c28feb4f1f04e1a8acb481d8596bc49, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=550, earliestPutTs=1681034096447 2023-04-09 09:54:58,464 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting b748fb1d35074d4fa307099a595ea43c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=562, earliestPutTs=1681034097571 2023-04-09 09:54:58,473 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] throttle.PressureAwareThroughputController(145): 026e485e431b415e1bb021c3d14de88b#A#compaction#306 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:54:58,475 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): 026e485e431b415e1bb021c3d14de88b#B#compaction#307 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:54:58,489 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/A/936d3f46463b4fef976825308c3ed0f7 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/936d3f46463b4fef976825308c3ed0f7 2023-04-09 09:54:58,493 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 026e485e431b415e1bb021c3d14de88b/A of 026e485e431b415e1bb021c3d14de88b into 936d3f46463b4fef976825308c3ed0f7(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:54:58,493 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 026e485e431b415e1bb021c3d14de88b: 2023-04-09 09:54:58,493 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b., storeName=026e485e431b415e1bb021c3d14de88b/A, priority=10, startTime=1681034098459; duration=0sec 2023-04-09 09:54:58,493 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:54:58,493 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-09 09:54:58,495 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71490 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-09 09:54:58,495 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1912): 026e485e431b415e1bb021c3d14de88b/C is initiating minor compaction (all files) 2023-04-09 09:54:58,495 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 026e485e431b415e1bb021c3d14de88b/C in TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:58,495 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/bdee1af652c040af8461c11efba78d10, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/1bb06a59bcfc47e0bdb4b891ddcfbbfb, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/2ae74df5615a47bb894c62dcdc3184a5, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/299a0beab0a7495da18fd0be015886bc, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/08525928d962485fb1f7c4ab5ee2b0ef, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/bf33f8604b2b42cbb842fc65f57d08b6] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp, totalSize=69.8 K 2023-04-09 09:54:58,495 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting bdee1af652c040af8461c11efba78d10, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=482, earliestPutTs=1681034093432 2023-04-09 09:54:58,496 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 1bb06a59bcfc47e0bdb4b891ddcfbbfb, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=497, earliestPutTs=1681034094559 2023-04-09 09:54:58,496 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 2ae74df5615a47bb894c62dcdc3184a5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=518, earliestPutTs=1681034095179 2023-04-09 09:54:58,496 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 299a0beab0a7495da18fd0be015886bc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=533, earliestPutTs=1681034095827 2023-04-09 09:54:58,497 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 08525928d962485fb1f7c4ab5ee2b0ef, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=550, earliestPutTs=1681034096447 2023-04-09 09:54:58,497 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting bf33f8604b2b42cbb842fc65f57d08b6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=562, earliestPutTs=1681034097571 2023-04-09 09:54:58,505 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] throttle.PressureAwareThroughputController(145): 026e485e431b415e1bb021c3d14de88b#C#compaction#308 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:54:58,513 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/C/6504dc6989ad44a08fd43e651f1835ad as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/6504dc6989ad44a08fd43e651f1835ad 2023-04-09 09:54:58,516 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 026e485e431b415e1bb021c3d14de88b/C of 026e485e431b415e1bb021c3d14de88b into 6504dc6989ad44a08fd43e651f1835ad(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:54:58,516 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 026e485e431b415e1bb021c3d14de88b: 2023-04-09 09:54:58,516 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b., storeName=026e485e431b415e1bb021c3d14de88b/C, priority=10, startTime=1681034098459; duration=0sec 2023-04-09 09:54:58,517 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:54:58,527 DEBUG [Thread-1715] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:58,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:58,529 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:58,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:58,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:58,530 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:58,530 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:58,530 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:58,530 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:58,531 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:58,531 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:58,532 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:58,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:58,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:58,532 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:58,533 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,533 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:58,533 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:58,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:58,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:58,533 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:58,533 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:58,533 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:58,533 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:58,533 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:58,533 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:58,534 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:58,534 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:58,534 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:58,534 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. started... 2023-04-09 09:54:58,534 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 026e485e431b415e1bb021c3d14de88b 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-09 09:54:58,534 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=A 2023-04-09 09:54:58,534 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:54:58,534 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=B 2023-04-09 09:54:58,534 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:54:58,534 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=C 2023-04-09 09:54:58,534 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:54:58,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 026e485e431b415e1bb021c3d14de88b 2023-04-09 09:54:58,719 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. as already flushing 2023-04-09 09:54:58,736 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:58,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56098 deadline: 1681034158733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,737 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:58,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56106 deadline: 1681034158737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,760 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:58,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56138 deadline: 1681034158757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,760 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:58,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 332 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56126 deadline: 1681034158757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,839 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:58,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56106 deadline: 1681034158838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,842 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:58,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56098 deadline: 1681034158839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:58,890 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/B/2e4028edb01e4b07aec7d74d6e0bd19a as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/2e4028edb01e4b07aec7d74d6e0bd19a 2023-04-09 09:54:58,894 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 026e485e431b415e1bb021c3d14de88b/B of 026e485e431b415e1bb021c3d14de88b into 2e4028edb01e4b07aec7d74d6e0bd19a(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:54:58,894 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 026e485e431b415e1bb021c3d14de88b: 2023-04-09 09:54:58,894 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b., storeName=026e485e431b415e1bb021c3d14de88b/B, priority=10, startTime=1681034098459; duration=0sec 2023-04-09 09:54:58,894 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:54:58,942 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=599 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/A/b0b611af73de4b3ab50adc2958a293be 2023-04-09 09:54:59,041 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:59,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56106 deadline: 1681034159041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:59,045 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:59,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56098 deadline: 1681034159044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:59,345 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:59,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56106 deadline: 1681034159344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:59,349 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:59,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56098 deadline: 1681034159347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:59,352 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=599 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/B/c1652adf7e2443f9a8b1e376955ca33f 2023-04-09 09:54:59,365 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=599 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/C/2fbf00d2147f4a77ad7ae3b00c874a03 2023-04-09 09:54:59,374 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/A/b0b611af73de4b3ab50adc2958a293be as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/b0b611af73de4b3ab50adc2958a293be 2023-04-09 09:54:59,378 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/b0b611af73de4b3ab50adc2958a293be, entries=150, sequenceid=599, filesize=11.9 K 2023-04-09 09:54:59,379 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/B/c1652adf7e2443f9a8b1e376955ca33f as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/c1652adf7e2443f9a8b1e376955ca33f 2023-04-09 09:54:59,383 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/c1652adf7e2443f9a8b1e376955ca33f, entries=150, sequenceid=599, filesize=11.9 K 2023-04-09 09:54:59,384 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/C/2fbf00d2147f4a77ad7ae3b00c874a03 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/2fbf00d2147f4a77ad7ae3b00c874a03 2023-04-09 09:54:59,387 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/2fbf00d2147f4a77ad7ae3b00c874a03, entries=150, sequenceid=599, filesize=11.9 K 2023-04-09 09:54:59,388 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for 026e485e431b415e1bb021c3d14de88b in 854ms, sequenceid=599, compaction requested=false 2023-04-09 09:54:59,388 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 026e485e431b415e1bb021c3d14de88b: 2023-04-09 09:54:59,388 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:59,388 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:54:59,388 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:54:59,388 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:59,388 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:54:59,388 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:54:59,391 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:59,391 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:59,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:59,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:59,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:59,391 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:59,391 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:54:59,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:59,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:59,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:59,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:59,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:59,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:54:59,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@244f186c[Count = 0] remaining members to acquire global barrier 2023-04-09 09:54:59,393 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:54:59,393 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:59,394 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:59,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:59,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:59,394 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:54:59,394 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:54:59,394 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:54:59,394 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:59,394 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:54:59,395 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:59,395 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:54:59,395 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:59,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:59,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:59,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:59,395 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:54:59,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:59,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:59,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:59,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:59,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:59,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:59,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:59,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:54:59,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:54:59,398 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:54:59,398 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:54:59,398 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:54:59,398 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:54:59,398 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:54:59,400 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:59,400 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:59,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:59,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:54:59,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:54:59,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:59,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:59,400 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:59,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:54:59,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:59,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:59,400 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:59,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:59,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:59,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:54:59,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:59,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:59,402 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:59,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:54:59,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:54:59,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:59,407 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:59,407 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:59,408 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:59,408 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:59,408 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:59,408 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:54:59,408 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:59,408 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:54:59,408 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:54:59,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:54:59,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:54:59,408 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:59,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:59,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:59,408 DEBUG [Thread-1715] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:54:59,408 DEBUG [Thread-1715] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:54:59,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:54:59,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:54:59,508 DEBUG [Thread-1715] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:54:59,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:54:59,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:54:59,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:54:59,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:54:59,512 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:54:59,512 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:59,512 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:54:59,512 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:54:59,512 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:59,512 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:59,513 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:54:59,513 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:59,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:54:59,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:54:59,514 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:59,514 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:54:59,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:59,514 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:54:59,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:54:59,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:54:59,514 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:54:59,515 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:54:59,515 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:54:59,515 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:54:59,515 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:54:59,515 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:54:59,515 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:54:59,515 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:54:59,515 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:54:59,515 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. started... 2023-04-09 09:54:59,515 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 026e485e431b415e1bb021c3d14de88b 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-09 09:54:59,515 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=A 2023-04-09 09:54:59,515 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:59,516 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=B 2023-04-09 09:54:59,516 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:59,516 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=C 2023-04-09 09:54:59,516 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:54:59,524 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=611 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/A/d3ff5d5709ea4c88a514992f12ad98cc 2023-04-09 09:54:59,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 026e485e431b415e1bb021c3d14de88b 2023-04-09 09:54:59,763 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. as already flushing 2023-04-09 09:54:59,788 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:59,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 344 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56126 deadline: 1681034159785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:59,788 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034099774, max timestamp=1681034099786Num uniques 859; 2023-04-09 09:54:59,788 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:54:59,788 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034099774, max timestamp=1681034099786Num uniques 859; 2023-04-09 09:54:59,788 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:54:59,788 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034099774, max timestamp=1681034099786Num uniques 859; 2023-04-09 09:54:59,788 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:54:59,832 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:59,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56138 deadline: 1681034159830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:59,837 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:59,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56168 deadline: 1681034159836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:59,839 DEBUG [Thread-1709] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18202 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b., hostname=jenkins-hbase4.apache.org,42467,1681034019544, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-09 09:54:59,852 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:59,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56106 deadline: 1681034159850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:59,852 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:59,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56098 deadline: 1681034159851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:59,891 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:59,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 346 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56126 deadline: 1681034159889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:54:59,933 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=611 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/B/039665eea4e74a659686c96bf24430fa 2023-04-09 09:54:59,935 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:54:59,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56138 deadline: 1681034159933, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,094 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:00,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 348 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56126 deadline: 1681034160093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,138 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:00,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56138 deadline: 1681034160137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,346 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=611 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/C/9b0f65f96cbd4d00b92a5cd600b5164a 2023-04-09 09:55:00,350 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/A/d3ff5d5709ea4c88a514992f12ad98cc as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/d3ff5d5709ea4c88a514992f12ad98cc 2023-04-09 09:55:00,353 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/d3ff5d5709ea4c88a514992f12ad98cc, entries=150, sequenceid=611, filesize=11.9 K 2023-04-09 09:55:00,354 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/B/039665eea4e74a659686c96bf24430fa as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/039665eea4e74a659686c96bf24430fa 2023-04-09 09:55:00,357 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/039665eea4e74a659686c96bf24430fa, entries=150, sequenceid=611, filesize=11.9 K 2023-04-09 09:55:00,358 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/C/9b0f65f96cbd4d00b92a5cd600b5164a as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/9b0f65f96cbd4d00b92a5cd600b5164a 2023-04-09 09:55:00,361 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/9b0f65f96cbd4d00b92a5cd600b5164a, entries=150, sequenceid=611, filesize=11.9 K 2023-04-09 09:55:00,362 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 026e485e431b415e1bb021c3d14de88b in 846ms, sequenceid=611, compaction requested=true 2023-04-09 09:55:00,362 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 026e485e431b415e1bb021c3d14de88b: 2023-04-09 09:55:00,362 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:55:00,362 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:00,362 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:00,362 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,362 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:00,362 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:00,363 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:00,363 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:00,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:00,364 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:00,364 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:00,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:00,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:00,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:00,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:00,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:00,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26881b52[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:00,365 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:00,365 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:00,367 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:00,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:00,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:00,367 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:00,367 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:00,367 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:00,367 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,367 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:00,369 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,369 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:00,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:00,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:00,369 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:00,369 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:00,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:00,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:00,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:00,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:00,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:00,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:00,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:00,371 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:00,371 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:00,371 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:00,371 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:00,371 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:00,373 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:00,373 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:00,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:00,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:00,373 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:00,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:00,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:00,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:00,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:00,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:00,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:00,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:00,374 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:00,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:00,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:00,379 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:00,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:00,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,381 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:00,381 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:00,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:00,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:00,382 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,382 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:00,382 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:00,382 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,382 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:00,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:00,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:00,382 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:00,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:00,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:00,382 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:00,382 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:00,382 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:00,382 DEBUG [Thread-1715] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:00,382 DEBUG [Thread-1715] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:00,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 026e485e431b415e1bb021c3d14de88b 2023-04-09 09:55:00,399 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 026e485e431b415e1bb021c3d14de88b 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-09 09:55:00,400 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=A 2023-04-09 09:55:00,400 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:55:00,400 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=B 2023-04-09 09:55:00,400 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:55:00,400 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=C 2023-04-09 09:55:00,400 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:55:00,425 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:00,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 358 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56126 deadline: 1681034160422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,442 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:00,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56138 deadline: 1681034160441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,483 DEBUG [Thread-1715] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:00,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:00,485 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:00,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:00,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:00,486 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:00,486 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:00,486 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:00,486 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:00,486 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:00,486 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:00,488 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:00,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:00,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:00,488 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,488 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,488 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:00,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:00,491 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:00,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:00,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:00,492 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:00,494 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:00,494 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:00,494 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:00,494 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:00,494 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:00,494 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:00,494 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:00,494 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:55:00,494 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. started... 2023-04-09 09:55:00,494 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. as already flushing 2023-04-09 09:55:00,495 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:55:00,495 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:00,495 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:00,495 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,495 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:00,495 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:00,497 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:00,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:00,497 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:00,497 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:00,497 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:00,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:00,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:00,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:00,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:00,499 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:00,499 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a142d6c[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:00,499 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:00,499 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:00,500 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:00,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:00,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:00,500 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:00,500 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:00,500 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,500 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:00,500 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:00,501 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,502 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:00,502 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:00,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:00,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:00,502 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:00,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:00,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:00,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:00,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:00,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:00,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:00,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:00,504 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:00,504 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:00,504 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:00,504 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:00,504 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:00,506 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:00,506 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:00,506 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:00,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:00,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:00,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:00,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:00,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:00,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:00,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:00,506 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:00,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:00,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:00,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:00,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:00,508 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:00,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:00,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,511 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:00,511 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:00,511 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:00,511 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:00,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:00,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:00,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:00,511 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:00,511 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:00,511 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:00,511 DEBUG [Thread-1715] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:00,511 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,511 DEBUG [Thread-1715] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:00,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:00,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:00,511 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:00,512 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:00,512 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:00,526 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:00,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 360 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56126 deadline: 1681034160526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,612 DEBUG [Thread-1715] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:00,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:00,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:00,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:00,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:00,615 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:00,615 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:00,615 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:00,615 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:00,615 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:00,615 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:00,616 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:00,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:00,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:00,616 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:00,617 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,617 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:00,617 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:00,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:00,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:00,617 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:00,618 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:00,618 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:00,618 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:00,618 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:00,618 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:00,618 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:00,618 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:00,618 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:55:00,618 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. started... 2023-04-09 09:55:00,618 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. as already flushing 2023-04-09 09:55:00,618 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:55:00,618 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:00,619 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:00,619 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,619 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:00,619 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:00,620 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,620 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:00,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:00,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:00,620 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:00,620 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:00,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:00,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:00,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:00,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:00,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:00,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72080c71[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:00,623 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:00,623 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:00,624 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:00,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:00,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:00,624 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:00,624 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:00,624 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:00,624 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,625 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:00,626 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,626 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:00,626 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:00,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:00,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:00,627 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:00,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:00,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:00,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:00,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:00,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:00,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:00,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:00,630 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:00,630 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:00,630 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:00,630 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:00,630 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:00,632 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:00,632 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:00,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:00,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:00,632 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:00,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:00,633 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:00,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:00,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:00,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:00,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:00,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:00,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:00,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:00,635 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:00,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:00,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:00,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,638 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:00,638 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,638 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:00,638 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:00,638 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:00,638 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:00,638 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:00,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:00,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:00,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:00,638 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:00,639 DEBUG [Thread-1715] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:00,639 DEBUG [Thread-1715] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:00,639 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:00,639 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:00,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:00,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:00,639 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:00,729 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:00,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 362 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56126 deadline: 1681034160727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,739 DEBUG [Thread-1715] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:00,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:00,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:00,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:00,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:00,742 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:00,742 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:00,742 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:00,742 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:00,743 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:00,743 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:00,747 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,747 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:00,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:00,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:00,747 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,748 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:00,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:00,748 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:00,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:00,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:00,748 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:00,748 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:00,749 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:00,749 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:00,749 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:00,749 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:00,749 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:00,749 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:00,749 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:55:00,749 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. started... 2023-04-09 09:55:00,749 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. as already flushing 2023-04-09 09:55:00,749 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:55:00,749 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:00,749 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:00,749 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,749 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:00,749 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:00,751 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:00,751 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:00,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:00,751 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:00,751 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:00,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:00,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:00,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:00,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:00,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:00,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e58e624[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:00,752 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:00,752 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:00,754 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:00,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:00,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:00,754 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:00,754 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:00,754 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,754 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:00,754 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:00,755 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,755 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:00,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:00,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:00,755 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:00,756 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:00,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:00,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:00,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:00,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:00,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:00,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:00,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:00,757 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:00,758 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:00,758 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:00,758 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:00,758 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:00,760 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:00,760 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:00,760 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:00,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:00,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:00,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:00,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:00,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:00,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:00,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:00,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:00,761 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:00,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:00,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:00,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:00,762 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:00,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:00,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,764 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,764 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:00,764 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:00,764 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:00,764 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:00,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:00,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:00,764 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:00,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:00,764 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:00,764 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:00,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:00,765 DEBUG [Thread-1715] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:00,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:00,764 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:00,765 DEBUG [Thread-1715] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:00,765 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:00,809 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=648 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/A/28bf43118a2d411a9165d3885208197c 2023-04-09 09:55:00,863 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:00,864 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:00,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56106 deadline: 1681034160861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56098 deadline: 1681034160861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,865 DEBUG [Thread-1715] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:00,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:00,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:00,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:00,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:00,868 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:00,868 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:00,868 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:00,868 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:00,868 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:00,868 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:00,869 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,869 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:00,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:00,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:00,870 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,870 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:00,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:00,870 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:00,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:00,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:00,870 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:00,871 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:00,871 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:00,871 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:00,871 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:00,871 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:00,871 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:00,871 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:00,871 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:55:00,871 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. started... 2023-04-09 09:55:00,871 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. as already flushing 2023-04-09 09:55:00,871 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:55:00,871 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:00,871 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:00,871 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,871 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:00,871 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:00,873 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,873 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:00,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:00,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:00,873 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:00,873 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:00,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:00,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:00,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:00,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:00,874 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:00,874 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@486c3a36[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:00,874 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:00,874 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:00,880 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:00,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:00,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:00,881 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:00,881 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,881 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:00,881 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:00,881 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:00,882 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,882 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:00,882 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:00,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:00,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:00,882 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:00,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:00,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:00,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:00,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:00,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:00,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:00,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:00,885 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:00,885 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:00,885 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:00,885 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:00,885 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:00,886 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:00,886 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:00,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:00,887 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:00,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:00,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:00,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:00,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:00,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:00,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:00,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:00,887 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:00,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:00,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:00,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:00,888 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:00,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:00,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,891 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:00,891 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:00,891 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:00,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:00,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:00,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:00,891 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:00,891 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:00,891 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:00,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:00,891 DEBUG [Thread-1715] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:00,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:00,891 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,891 DEBUG [Thread-1715] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:00,891 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:00,892 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:00,892 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:00,948 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:00,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56138 deadline: 1681034160946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:00,961 DEBUG [Thread-1724] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6fcbaba8 to 127.0.0.1:57816 2023-04-09 09:55:00,961 DEBUG [Thread-1724] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-09 09:55:00,961 DEBUG [Thread-1718] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x087b3095 to 127.0.0.1:57816 2023-04-09 09:55:00,961 DEBUG [Thread-1718] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-09 09:55:00,963 DEBUG [Thread-1716] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2ce5573e to 127.0.0.1:57816 2023-04-09 09:55:00,963 DEBUG [Thread-1716] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-09 09:55:00,964 DEBUG [Thread-1720] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x12115052 to 127.0.0.1:57816 2023-04-09 09:55:00,964 DEBUG [Thread-1720] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-09 09:55:00,964 DEBUG [Thread-1722] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7e65578e to 127.0.0.1:57816 2023-04-09 09:55:00,964 DEBUG [Thread-1722] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-09 09:55:00,992 DEBUG [Thread-1715] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:00,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:01,031 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:01,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 364 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56126 deadline: 1681034161031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:01,223 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=648 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/B/a0cd7344b5e74f8d90d2c9dd59b95116 2023-04-09 09:55:01,537 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:01,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 366 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56126 deadline: 1681034161536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:01,632 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=648 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/C/c666d36c97e740509374fc9a1624148b 2023-04-09 09:55:01,636 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/A/28bf43118a2d411a9165d3885208197c as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/28bf43118a2d411a9165d3885208197c 2023-04-09 09:55:01,639 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/28bf43118a2d411a9165d3885208197c, entries=200, sequenceid=648, filesize=14.3 K 2023-04-09 09:55:01,640 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/B/a0cd7344b5e74f8d90d2c9dd59b95116 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/a0cd7344b5e74f8d90d2c9dd59b95116 2023-04-09 09:55:01,643 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/a0cd7344b5e74f8d90d2c9dd59b95116, entries=150, sequenceid=648, filesize=11.9 K 2023-04-09 09:55:01,643 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/C/c666d36c97e740509374fc9a1624148b as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/c666d36c97e740509374fc9a1624148b 2023-04-09 09:55:01,646 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/c666d36c97e740509374fc9a1624148b, entries=150, sequenceid=648, filesize=11.9 K 2023-04-09 09:55:01,647 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 026e485e431b415e1bb021c3d14de88b in 1248ms, sequenceid=648, compaction requested=true 2023-04-09 09:55:01,647 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 026e485e431b415e1bb021c3d14de88b: 2023-04-09 09:55:01,647 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:55:01,647 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:55:01,647 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-09 09:55:01,647 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-09 09:55:01,647 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:55:01,648 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 52206 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-09 09:55:01,648 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49766 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-09 09:55:01,648 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1912): 026e485e431b415e1bb021c3d14de88b/A is initiating minor compaction (all files) 2023-04-09 09:55:01,648 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): 026e485e431b415e1bb021c3d14de88b/B is initiating minor compaction (all files) 2023-04-09 09:55:01,648 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 026e485e431b415e1bb021c3d14de88b/A in TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:55:01,648 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 026e485e431b415e1bb021c3d14de88b/B in TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:55:01,649 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/936d3f46463b4fef976825308c3ed0f7, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/b0b611af73de4b3ab50adc2958a293be, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/d3ff5d5709ea4c88a514992f12ad98cc, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/28bf43118a2d411a9165d3885208197c] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp, totalSize=51.0 K 2023-04-09 09:55:01,649 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/2e4028edb01e4b07aec7d74d6e0bd19a, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/c1652adf7e2443f9a8b1e376955ca33f, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/039665eea4e74a659686c96bf24430fa, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/a0cd7344b5e74f8d90d2c9dd59b95116] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp, totalSize=48.6 K 2023-04-09 09:55:01,649 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 936d3f46463b4fef976825308c3ed0f7, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=562, earliestPutTs=1681034097571 2023-04-09 09:55:01,649 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 2e4028edb01e4b07aec7d74d6e0bd19a, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=562, earliestPutTs=1681034097571 2023-04-09 09:55:01,649 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting b0b611af73de4b3ab50adc2958a293be, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=599, earliestPutTs=1681034097626 2023-04-09 09:55:01,649 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting c1652adf7e2443f9a8b1e376955ca33f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=599, earliestPutTs=1681034097626 2023-04-09 09:55:01,649 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting d3ff5d5709ea4c88a514992f12ad98cc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=611, earliestPutTs=1681034098731 2023-04-09 09:55:01,649 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 039665eea4e74a659686c96bf24430fa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=611, earliestPutTs=1681034098731 2023-04-09 09:55:01,649 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 28bf43118a2d411a9165d3885208197c, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=648, earliestPutTs=1681034099824 2023-04-09 09:55:01,650 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting a0cd7344b5e74f8d90d2c9dd59b95116, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=648, earliestPutTs=1681034099824 2023-04-09 09:55:01,656 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] throttle.PressureAwareThroughputController(145): 026e485e431b415e1bb021c3d14de88b#A#compaction#318 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:55:01,657 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): 026e485e431b415e1bb021c3d14de88b#B#compaction#319 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:55:01,669 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/B/22af89e1eb16405b8966fa5c82f3df94 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/22af89e1eb16405b8966fa5c82f3df94 2023-04-09 09:55:01,673 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 026e485e431b415e1bb021c3d14de88b/B of 026e485e431b415e1bb021c3d14de88b into 22af89e1eb16405b8966fa5c82f3df94(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:55:01,673 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 026e485e431b415e1bb021c3d14de88b: 2023-04-09 09:55:01,673 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b., storeName=026e485e431b415e1bb021c3d14de88b/B, priority=12, startTime=1681034101647; duration=0sec 2023-04-09 09:55:01,673 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:55:01,673 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-09 09:55:01,674 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49766 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-09 09:55:01,674 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): 026e485e431b415e1bb021c3d14de88b/C is initiating minor compaction (all files) 2023-04-09 09:55:01,674 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 026e485e431b415e1bb021c3d14de88b/C in TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:55:01,674 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/6504dc6989ad44a08fd43e651f1835ad, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/2fbf00d2147f4a77ad7ae3b00c874a03, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/9b0f65f96cbd4d00b92a5cd600b5164a, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/c666d36c97e740509374fc9a1624148b] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp, totalSize=48.6 K 2023-04-09 09:55:01,674 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 6504dc6989ad44a08fd43e651f1835ad, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=562, earliestPutTs=1681034097571 2023-04-09 09:55:01,675 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 2fbf00d2147f4a77ad7ae3b00c874a03, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=599, earliestPutTs=1681034097626 2023-04-09 09:55:01,675 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 9b0f65f96cbd4d00b92a5cd600b5164a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=611, earliestPutTs=1681034098731 2023-04-09 09:55:01,675 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting c666d36c97e740509374fc9a1624148b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=648, earliestPutTs=1681034099824 2023-04-09 09:55:01,680 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): 026e485e431b415e1bb021c3d14de88b#C#compaction#320 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:55:01,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 026e485e431b415e1bb021c3d14de88b 2023-04-09 09:55:01,952 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 026e485e431b415e1bb021c3d14de88b 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-09 09:55:01,952 DEBUG [Thread-1705] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x262f4570 to 127.0.0.1:57816 2023-04-09 09:55:01,952 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=A 2023-04-09 09:55:01,952 DEBUG [Thread-1705] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-09 09:55:01,952 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:01,952 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=B 2023-04-09 09:55:01,952 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:01,952 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=C 2023-04-09 09:55:01,952 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:02,069 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/A/cd9fd046bc7747de898c93f2ed40819c as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/cd9fd046bc7747de898c93f2ed40819c 2023-04-09 09:55:02,073 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 026e485e431b415e1bb021c3d14de88b/A of 026e485e431b415e1bb021c3d14de88b into cd9fd046bc7747de898c93f2ed40819c(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:55:02,073 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 026e485e431b415e1bb021c3d14de88b: 2023-04-09 09:55:02,073 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b., storeName=026e485e431b415e1bb021c3d14de88b/A, priority=12, startTime=1681034101647; duration=0sec 2023-04-09 09:55:02,073 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:55:02,087 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/C/fcdb8c4f0a37490e99cf108d19fc145b as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/fcdb8c4f0a37490e99cf108d19fc145b 2023-04-09 09:55:02,091 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 026e485e431b415e1bb021c3d14de88b/C of 026e485e431b415e1bb021c3d14de88b into fcdb8c4f0a37490e99cf108d19fc145b(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:55:02,091 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 026e485e431b415e1bb021c3d14de88b: 2023-04-09 09:55:02,091 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b., storeName=026e485e431b415e1bb021c3d14de88b/C, priority=12, startTime=1681034101647; duration=0sec 2023-04-09 09:55:02,091 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:55:02,359 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=660 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/A/3ec276dd67d7402e820a9841325f0fe5 2023-04-09 09:55:02,540 DEBUG [Thread-1711] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7480f9da to 127.0.0.1:57816 2023-04-09 09:55:02,540 DEBUG [Thread-1711] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-09 09:55:02,768 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=660 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/B/c4a77b97982d4a0899d49ea4d573d9ec 2023-04-09 09:55:02,876 DEBUG [Thread-1707] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x00ca898c to 127.0.0.1:57816 2023-04-09 09:55:02,876 DEBUG [Thread-1707] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-09 09:55:02,881 DEBUG [Thread-1713] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1a24d617 to 127.0.0.1:57816 2023-04-09 09:55:02,881 DEBUG [Thread-1713] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-09 09:55:03,177 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=660 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/C/b28f29cede974017a86d114b27817917 2023-04-09 09:55:03,180 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/A/3ec276dd67d7402e820a9841325f0fe5 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/3ec276dd67d7402e820a9841325f0fe5 2023-04-09 09:55:03,183 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/3ec276dd67d7402e820a9841325f0fe5, entries=150, sequenceid=660, filesize=11.9 K 2023-04-09 09:55:03,184 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/B/c4a77b97982d4a0899d49ea4d573d9ec as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/c4a77b97982d4a0899d49ea4d573d9ec 2023-04-09 09:55:03,187 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/c4a77b97982d4a0899d49ea4d573d9ec, entries=150, sequenceid=660, filesize=11.9 K 2023-04-09 09:55:03,187 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/C/b28f29cede974017a86d114b27817917 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/b28f29cede974017a86d114b27817917 2023-04-09 09:55:03,191 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/b28f29cede974017a86d114b27817917, entries=150, sequenceid=660, filesize=11.9 K 2023-04-09 09:55:03,192 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=20.13 KB/20610 for 026e485e431b415e1bb021c3d14de88b in 1239ms, sequenceid=660, compaction requested=false 2023-04-09 09:55:03,192 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 026e485e431b415e1bb021c3d14de88b: 2023-04-09 09:55:09,905 DEBUG [Thread-1709] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6585a70a to 127.0.0.1:57816 2023-04-09 09:55:09,905 DEBUG [Thread-1709] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-09 09:55:09,905 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-09 09:55:09,905 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(394): wrote 159 2023-04-09 09:55:09,905 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(394): wrote 96 2023-04-09 09:55:09,905 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(394): wrote 7 2023-04-09 09:55:09,906 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(394): wrote 209 2023-04-09 09:55:09,906 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(394): wrote 85 2023-04-09 09:55:09,906 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-09 09:55:09,906 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(398): read 5306 2023-04-09 09:55:09,906 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(398): read 5150 2023-04-09 09:55:09,906 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(398): read 5171 2023-04-09 09:55:09,906 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(398): read 5276 2023-04-09 09:55:09,906 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(398): read 5151 2023-04-09 09:55:09,906 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-09 09:55:09,906 INFO [Listener at localhost/44563] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-09 09:55:09,906 DEBUG [Listener at localhost/44563] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x22b4eae5 to 127.0.0.1:57816 2023-04-09 09:55:09,906 DEBUG [Listener at localhost/44563] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-09 09:55:09,907 INFO [Listener at localhost/44563] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-09 09:55:09,907 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-09 09:55:09,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure2.ProcedureExecutor(1029): Stored pid=31, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-09 09:55:09,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-09 09:55:09,910 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681034109910"}]},"ts":"1681034109910"} 2023-04-09 09:55:09,911 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-09 09:55:09,912 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-09 09:55:09,913 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=026e485e431b415e1bb021c3d14de88b, UNASSIGN}] 2023-04-09 09:55:09,914 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=026e485e431b415e1bb021c3d14de88b, UNASSIGN 2023-04-09 09:55:09,914 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=026e485e431b415e1bb021c3d14de88b, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:09,915 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681034109914"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681034109914"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681034109914"}]},"ts":"1681034109914"} 2023-04-09 09:55:09,915 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=32, state=RUNNABLE; CloseRegionProcedure 026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544}] 2023-04-09 09:55:10,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-09 09:55:10,067 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 026e485e431b415e1bb021c3d14de88b 2023-04-09 09:55:10,067 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 026e485e431b415e1bb021c3d14de88b, disabling compactions & flushes 2023-04-09 09:55:10,067 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:55:10,067 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:55:10,067 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. after waiting 0 ms 2023-04-09 09:55:10,067 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:55:10,067 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 026e485e431b415e1bb021c3d14de88b 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-09 09:55:10,068 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=A 2023-04-09 09:55:10,068 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:10,068 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=B 2023-04-09 09:55:10,068 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:10,068 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 026e485e431b415e1bb021c3d14de88b, store=C 2023-04-09 09:55:10,068 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:10,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-09 09:55:10,475 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=669 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/A/f6d7972eb037448d99e0a0ffbabf7a0e 2023-04-09 09:55:10,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-09 09:55:10,884 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=669 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/B/80e7dcbecdbd4a61840173a2aefe61c5 2023-04-09 09:55:11,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-09 09:55:11,292 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=669 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/C/f8fc755cc0844c60b88e5e93748b6d15 2023-04-09 09:55:11,296 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/A/f6d7972eb037448d99e0a0ffbabf7a0e as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/f6d7972eb037448d99e0a0ffbabf7a0e 2023-04-09 09:55:11,299 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/f6d7972eb037448d99e0a0ffbabf7a0e, entries=100, sequenceid=669, filesize=9.5 K 2023-04-09 09:55:11,300 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/B/80e7dcbecdbd4a61840173a2aefe61c5 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/80e7dcbecdbd4a61840173a2aefe61c5 2023-04-09 09:55:11,303 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/80e7dcbecdbd4a61840173a2aefe61c5, entries=100, sequenceid=669, filesize=9.5 K 2023-04-09 09:55:11,303 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/.tmp/C/f8fc755cc0844c60b88e5e93748b6d15 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/f8fc755cc0844c60b88e5e93748b6d15 2023-04-09 09:55:11,306 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/f8fc755cc0844c60b88e5e93748b6d15, entries=100, sequenceid=669, filesize=9.5 K 2023-04-09 09:55:11,307 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 026e485e431b415e1bb021c3d14de88b in 1240ms, sequenceid=669, compaction requested=true 2023-04-09 09:55:11,329 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/f502705a26b545028f6d111a3e52184f, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/57bb5d34e0b246c78e148df37acf6353, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/1ef21a481a224b2a992ee1e3e0a1d751, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/608b0420f8ef4396958cfcf2ba6b8b67, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/2622df9731e54b299d0370686a2ba6c0, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/c2f492d8b26140288a2859a72d3f9fd8, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/f57fb7e9664445deb2c2a5a9bc067f27, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/c3866e6c1489427c9bc6d2f16462952a, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/a893115462b04945b4d93807770c6c3a, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/b034770880d242788150337c0cc951c4, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/40e3c6055f294144bbd76a30e0f64eaa, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/c4e1ed9274f740ccafcf844c36f81e2d, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/dc633494d4f54c7f961912c1c3641301, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/c3f60e7cb514480aaf7b6075ffee4bdb, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/bbea8e3cc6c04a988444e4e62cfd4423, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/7fc5b1bac4fc450f89fffa73aef02658, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/681601aab7644da79a4f1a9bd5d3e798, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/58847ab295994ed5b8b805376c7dee2e, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/5e607fee10024a0580f41cd46797a436, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/ea78ac69f98e4f8e84169e39aad7ba50, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/6fd0c9634496420b87fa599434913b92, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/b1d06e51876947fdade72cfe2316b54c, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/1d874aaf37f64d0984d90bdebb60651a, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/2cd94d187dac4bdabcfe41f807dff3ff, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/23b2d331ad1347d2b7d89fb7fe5f4b69, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/1c0ac7d6c32a4087a6fa09be413d302d, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/10e74cf2ce0d4f2ea9976a89950bd8b9, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/fdccd7f13a6840a1afc0e443aac49aae, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/20751b05a1d14e969751dc0161360aa2, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/a4dfdfac409f49a1aa95b2beb45b6222, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/936d3f46463b4fef976825308c3ed0f7, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/b0b611af73de4b3ab50adc2958a293be, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/d3ff5d5709ea4c88a514992f12ad98cc, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/28bf43118a2d411a9165d3885208197c] to archive 2023-04-09 09:55:11,330 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-09 09:55:11,332 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/f502705a26b545028f6d111a3e52184f to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/f502705a26b545028f6d111a3e52184f 2023-04-09 09:55:11,333 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/57bb5d34e0b246c78e148df37acf6353 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/57bb5d34e0b246c78e148df37acf6353 2023-04-09 09:55:11,334 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/1ef21a481a224b2a992ee1e3e0a1d751 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/1ef21a481a224b2a992ee1e3e0a1d751 2023-04-09 09:55:11,335 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/608b0420f8ef4396958cfcf2ba6b8b67 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/608b0420f8ef4396958cfcf2ba6b8b67 2023-04-09 09:55:11,335 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/2622df9731e54b299d0370686a2ba6c0 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/2622df9731e54b299d0370686a2ba6c0 2023-04-09 09:55:11,336 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/c2f492d8b26140288a2859a72d3f9fd8 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/c2f492d8b26140288a2859a72d3f9fd8 2023-04-09 09:55:11,337 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/f57fb7e9664445deb2c2a5a9bc067f27 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/f57fb7e9664445deb2c2a5a9bc067f27 2023-04-09 09:55:11,338 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/c3866e6c1489427c9bc6d2f16462952a to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/c3866e6c1489427c9bc6d2f16462952a 2023-04-09 09:55:11,339 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/a893115462b04945b4d93807770c6c3a to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/a893115462b04945b4d93807770c6c3a 2023-04-09 09:55:11,339 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/b034770880d242788150337c0cc951c4 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/b034770880d242788150337c0cc951c4 2023-04-09 09:55:11,341 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/40e3c6055f294144bbd76a30e0f64eaa to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/40e3c6055f294144bbd76a30e0f64eaa 2023-04-09 09:55:11,342 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/c4e1ed9274f740ccafcf844c36f81e2d to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/c4e1ed9274f740ccafcf844c36f81e2d 2023-04-09 09:55:11,342 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/dc633494d4f54c7f961912c1c3641301 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/dc633494d4f54c7f961912c1c3641301 2023-04-09 09:55:11,344 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/c3f60e7cb514480aaf7b6075ffee4bdb to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/c3f60e7cb514480aaf7b6075ffee4bdb 2023-04-09 09:55:11,344 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/bbea8e3cc6c04a988444e4e62cfd4423 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/bbea8e3cc6c04a988444e4e62cfd4423 2023-04-09 09:55:11,345 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/7fc5b1bac4fc450f89fffa73aef02658 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/7fc5b1bac4fc450f89fffa73aef02658 2023-04-09 09:55:11,346 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/681601aab7644da79a4f1a9bd5d3e798 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/681601aab7644da79a4f1a9bd5d3e798 2023-04-09 09:55:11,347 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/58847ab295994ed5b8b805376c7dee2e to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/58847ab295994ed5b8b805376c7dee2e 2023-04-09 09:55:11,348 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/5e607fee10024a0580f41cd46797a436 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/5e607fee10024a0580f41cd46797a436 2023-04-09 09:55:11,348 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/ea78ac69f98e4f8e84169e39aad7ba50 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/ea78ac69f98e4f8e84169e39aad7ba50 2023-04-09 09:55:11,349 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/6fd0c9634496420b87fa599434913b92 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/6fd0c9634496420b87fa599434913b92 2023-04-09 09:55:11,350 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/b1d06e51876947fdade72cfe2316b54c to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/b1d06e51876947fdade72cfe2316b54c 2023-04-09 09:55:11,351 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/1d874aaf37f64d0984d90bdebb60651a to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/1d874aaf37f64d0984d90bdebb60651a 2023-04-09 09:55:11,352 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/2cd94d187dac4bdabcfe41f807dff3ff to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/2cd94d187dac4bdabcfe41f807dff3ff 2023-04-09 09:55:11,352 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/23b2d331ad1347d2b7d89fb7fe5f4b69 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/23b2d331ad1347d2b7d89fb7fe5f4b69 2023-04-09 09:55:11,353 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/1c0ac7d6c32a4087a6fa09be413d302d to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/1c0ac7d6c32a4087a6fa09be413d302d 2023-04-09 09:55:11,354 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/10e74cf2ce0d4f2ea9976a89950bd8b9 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/10e74cf2ce0d4f2ea9976a89950bd8b9 2023-04-09 09:55:11,355 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/fdccd7f13a6840a1afc0e443aac49aae to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/fdccd7f13a6840a1afc0e443aac49aae 2023-04-09 09:55:11,356 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/20751b05a1d14e969751dc0161360aa2 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/20751b05a1d14e969751dc0161360aa2 2023-04-09 09:55:11,357 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/a4dfdfac409f49a1aa95b2beb45b6222 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/a4dfdfac409f49a1aa95b2beb45b6222 2023-04-09 09:55:11,357 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/936d3f46463b4fef976825308c3ed0f7 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/936d3f46463b4fef976825308c3ed0f7 2023-04-09 09:55:11,358 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/b0b611af73de4b3ab50adc2958a293be to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/b0b611af73de4b3ab50adc2958a293be 2023-04-09 09:55:11,359 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/d3ff5d5709ea4c88a514992f12ad98cc to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/d3ff5d5709ea4c88a514992f12ad98cc 2023-04-09 09:55:11,360 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/28bf43118a2d411a9165d3885208197c to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/28bf43118a2d411a9165d3885208197c 2023-04-09 09:55:11,363 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/a256341ad74447948ce54982a841198c, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/a2ff42ee6fe94b72a1ab2dce64120332, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/d99c438e56864157a835fb143ad7a8aa, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/120fd20790654e6698fecd9d6728c6ea, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/d05195cb2cb94afeb94d7ca6494119fb, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/acf69788d8ef43eea271b334d0105ff6, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/196396ca09244474bddbb4bbe6a5b6e4, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/6db091ddcac54a84a2b8fa9921b85016, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/5eefd7952e6f494bbddf84b13487ebb2, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/713ce4b41b3348b78d88385a5fce2c4c, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/4b0abea584904282ae9d1ba6126ec9e1, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/e29f3b54a2264a4db673044816da1138, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/9bc9c819a95745f9a4716363ab4321a1, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/e6dcaf082c7846829bbf2ea146929c29, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/d3ac8699bc2940179c8d4f0f78d80a16, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/93235309e4db4d908ad350ad8ce89fff, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/8dd1a7e0543b4e3da9c5b8227f58a6c7, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/ed8f2fbee1784e998f9951b81f43ef2b, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/f5cb4f45579a4c9289d52bc94c37b8cd, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/abcfc87f65874425a6dfef508f0e91a5, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/06759505757b44b4ae6ae5b90cfde65c, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/4a1492c05b564fc5a8f95c0d2bad988a, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/d2f6bc62c786484db0e1e1e647427358, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/f0859b7198294b4d9dfa830650db9cc4, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/597dcfacd1104df3b98a2e9c27a0cdc0, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/d36a0008f5c0482c9da4f8984fd87caa, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/5ced19337b04406cace2075d31611bbf, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/44943f53ea9c440fad45f9c9df9ea211, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/1c28feb4f1f04e1a8acb481d8596bc49, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/2e4028edb01e4b07aec7d74d6e0bd19a, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/b748fb1d35074d4fa307099a595ea43c, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/c1652adf7e2443f9a8b1e376955ca33f, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/039665eea4e74a659686c96bf24430fa, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/a0cd7344b5e74f8d90d2c9dd59b95116] to archive 2023-04-09 09:55:11,364 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-09 09:55:11,365 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/a256341ad74447948ce54982a841198c to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/a256341ad74447948ce54982a841198c 2023-04-09 09:55:11,366 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/a2ff42ee6fe94b72a1ab2dce64120332 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/a2ff42ee6fe94b72a1ab2dce64120332 2023-04-09 09:55:11,367 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/d99c438e56864157a835fb143ad7a8aa to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/d99c438e56864157a835fb143ad7a8aa 2023-04-09 09:55:11,367 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/120fd20790654e6698fecd9d6728c6ea to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/120fd20790654e6698fecd9d6728c6ea 2023-04-09 09:55:11,368 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/d05195cb2cb94afeb94d7ca6494119fb to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/d05195cb2cb94afeb94d7ca6494119fb 2023-04-09 09:55:11,369 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/acf69788d8ef43eea271b334d0105ff6 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/acf69788d8ef43eea271b334d0105ff6 2023-04-09 09:55:11,370 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/196396ca09244474bddbb4bbe6a5b6e4 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/196396ca09244474bddbb4bbe6a5b6e4 2023-04-09 09:55:11,371 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/6db091ddcac54a84a2b8fa9921b85016 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/6db091ddcac54a84a2b8fa9921b85016 2023-04-09 09:55:11,372 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/5eefd7952e6f494bbddf84b13487ebb2 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/5eefd7952e6f494bbddf84b13487ebb2 2023-04-09 09:55:11,373 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/713ce4b41b3348b78d88385a5fce2c4c to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/713ce4b41b3348b78d88385a5fce2c4c 2023-04-09 09:55:11,374 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/4b0abea584904282ae9d1ba6126ec9e1 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/4b0abea584904282ae9d1ba6126ec9e1 2023-04-09 09:55:11,375 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/e29f3b54a2264a4db673044816da1138 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/e29f3b54a2264a4db673044816da1138 2023-04-09 09:55:11,375 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/9bc9c819a95745f9a4716363ab4321a1 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/9bc9c819a95745f9a4716363ab4321a1 2023-04-09 09:55:11,376 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/e6dcaf082c7846829bbf2ea146929c29 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/e6dcaf082c7846829bbf2ea146929c29 2023-04-09 09:55:11,377 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/d3ac8699bc2940179c8d4f0f78d80a16 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/d3ac8699bc2940179c8d4f0f78d80a16 2023-04-09 09:55:11,378 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/93235309e4db4d908ad350ad8ce89fff to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/93235309e4db4d908ad350ad8ce89fff 2023-04-09 09:55:11,379 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/8dd1a7e0543b4e3da9c5b8227f58a6c7 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/8dd1a7e0543b4e3da9c5b8227f58a6c7 2023-04-09 09:55:11,380 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/ed8f2fbee1784e998f9951b81f43ef2b to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/ed8f2fbee1784e998f9951b81f43ef2b 2023-04-09 09:55:11,380 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/f5cb4f45579a4c9289d52bc94c37b8cd to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/f5cb4f45579a4c9289d52bc94c37b8cd 2023-04-09 09:55:11,381 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/abcfc87f65874425a6dfef508f0e91a5 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/abcfc87f65874425a6dfef508f0e91a5 2023-04-09 09:55:11,382 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/06759505757b44b4ae6ae5b90cfde65c to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/06759505757b44b4ae6ae5b90cfde65c 2023-04-09 09:55:11,383 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/4a1492c05b564fc5a8f95c0d2bad988a to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/4a1492c05b564fc5a8f95c0d2bad988a 2023-04-09 09:55:11,384 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/d2f6bc62c786484db0e1e1e647427358 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/d2f6bc62c786484db0e1e1e647427358 2023-04-09 09:55:11,384 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/f0859b7198294b4d9dfa830650db9cc4 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/f0859b7198294b4d9dfa830650db9cc4 2023-04-09 09:55:11,385 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/597dcfacd1104df3b98a2e9c27a0cdc0 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/597dcfacd1104df3b98a2e9c27a0cdc0 2023-04-09 09:55:11,386 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/d36a0008f5c0482c9da4f8984fd87caa to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/d36a0008f5c0482c9da4f8984fd87caa 2023-04-09 09:55:11,387 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/5ced19337b04406cace2075d31611bbf to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/5ced19337b04406cace2075d31611bbf 2023-04-09 09:55:11,387 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/44943f53ea9c440fad45f9c9df9ea211 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/44943f53ea9c440fad45f9c9df9ea211 2023-04-09 09:55:11,388 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/1c28feb4f1f04e1a8acb481d8596bc49 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/1c28feb4f1f04e1a8acb481d8596bc49 2023-04-09 09:55:11,389 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/2e4028edb01e4b07aec7d74d6e0bd19a to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/2e4028edb01e4b07aec7d74d6e0bd19a 2023-04-09 09:55:11,390 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/b748fb1d35074d4fa307099a595ea43c to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/b748fb1d35074d4fa307099a595ea43c 2023-04-09 09:55:11,390 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/c1652adf7e2443f9a8b1e376955ca33f to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/c1652adf7e2443f9a8b1e376955ca33f 2023-04-09 09:55:11,391 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/039665eea4e74a659686c96bf24430fa to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/039665eea4e74a659686c96bf24430fa 2023-04-09 09:55:11,392 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/a0cd7344b5e74f8d90d2c9dd59b95116 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/a0cd7344b5e74f8d90d2c9dd59b95116 2023-04-09 09:55:11,395 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/4814e1f19e2e4ed48dde22c63190a534, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/246a21a132c942a29f36740d4561825e, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/20c157f0ff8b4cb496b16a85c08b55b2, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/b5f0f964faf24fb2955bc8056dc955d4, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/62f0b22ba5e94888b26c3ad48c981f28, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/0b2d5aa00056494b90135fb04ed7d5e0, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/d06e77d285e946f19aa3a65434c45337, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/db9800862dce40fdbc6a56ba303213a5, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/476073b5e2f9448e9b9730d4ea71d047, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/f38a2531c1144f3fae0c8a9ad08d16ed, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/5dd892ec49314fce963be17b17273d46, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/0e13428080fc4e679a89c03840b75971, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/53aee86ae46a4d048925d35254c620be, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/5803cc5d47804bedaa8725d35cb8a647, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/98c33ee6a374474bbe44ee5c7e8b3151, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/471df4ad1aa34cdbbca1049ebcb45cf1, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/ca5b2f2457f94ab2be06985611429bb1, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/2052ff9828184c84aa0abba422b4fcbe, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/1746922aba2f49a2bb62fd8ed5c97454, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/700fc3f06f134c4e8c192323af609ac0, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/9b25c1afba7c49f3a68633202362cd48, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/2fc52d3e940c430e829135e1b811783f, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/d5e7a4014a3d45f18d311c5096db71ea, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/bdee1af652c040af8461c11efba78d10, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/8e021e2fe2834b1e93ab5608740ca248, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/1bb06a59bcfc47e0bdb4b891ddcfbbfb, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/2ae74df5615a47bb894c62dcdc3184a5, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/299a0beab0a7495da18fd0be015886bc, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/08525928d962485fb1f7c4ab5ee2b0ef, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/6504dc6989ad44a08fd43e651f1835ad, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/bf33f8604b2b42cbb842fc65f57d08b6, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/2fbf00d2147f4a77ad7ae3b00c874a03, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/9b0f65f96cbd4d00b92a5cd600b5164a, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/c666d36c97e740509374fc9a1624148b] to archive 2023-04-09 09:55:11,395 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-09 09:55:11,396 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/4814e1f19e2e4ed48dde22c63190a534 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/4814e1f19e2e4ed48dde22c63190a534 2023-04-09 09:55:11,397 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/246a21a132c942a29f36740d4561825e to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/246a21a132c942a29f36740d4561825e 2023-04-09 09:55:11,398 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/20c157f0ff8b4cb496b16a85c08b55b2 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/20c157f0ff8b4cb496b16a85c08b55b2 2023-04-09 09:55:11,399 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/b5f0f964faf24fb2955bc8056dc955d4 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/b5f0f964faf24fb2955bc8056dc955d4 2023-04-09 09:55:11,399 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/62f0b22ba5e94888b26c3ad48c981f28 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/62f0b22ba5e94888b26c3ad48c981f28 2023-04-09 09:55:11,400 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/0b2d5aa00056494b90135fb04ed7d5e0 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/0b2d5aa00056494b90135fb04ed7d5e0 2023-04-09 09:55:11,401 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/d06e77d285e946f19aa3a65434c45337 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/d06e77d285e946f19aa3a65434c45337 2023-04-09 09:55:11,402 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/db9800862dce40fdbc6a56ba303213a5 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/db9800862dce40fdbc6a56ba303213a5 2023-04-09 09:55:11,402 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/476073b5e2f9448e9b9730d4ea71d047 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/476073b5e2f9448e9b9730d4ea71d047 2023-04-09 09:55:11,403 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/f38a2531c1144f3fae0c8a9ad08d16ed to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/f38a2531c1144f3fae0c8a9ad08d16ed 2023-04-09 09:55:11,404 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/5dd892ec49314fce963be17b17273d46 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/5dd892ec49314fce963be17b17273d46 2023-04-09 09:55:11,405 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/0e13428080fc4e679a89c03840b75971 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/0e13428080fc4e679a89c03840b75971 2023-04-09 09:55:11,406 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/53aee86ae46a4d048925d35254c620be to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/53aee86ae46a4d048925d35254c620be 2023-04-09 09:55:11,406 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/5803cc5d47804bedaa8725d35cb8a647 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/5803cc5d47804bedaa8725d35cb8a647 2023-04-09 09:55:11,407 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/98c33ee6a374474bbe44ee5c7e8b3151 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/98c33ee6a374474bbe44ee5c7e8b3151 2023-04-09 09:55:11,408 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/471df4ad1aa34cdbbca1049ebcb45cf1 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/471df4ad1aa34cdbbca1049ebcb45cf1 2023-04-09 09:55:11,409 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/ca5b2f2457f94ab2be06985611429bb1 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/ca5b2f2457f94ab2be06985611429bb1 2023-04-09 09:55:11,410 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/2052ff9828184c84aa0abba422b4fcbe to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/2052ff9828184c84aa0abba422b4fcbe 2023-04-09 09:55:11,410 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/1746922aba2f49a2bb62fd8ed5c97454 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/1746922aba2f49a2bb62fd8ed5c97454 2023-04-09 09:55:11,411 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/700fc3f06f134c4e8c192323af609ac0 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/700fc3f06f134c4e8c192323af609ac0 2023-04-09 09:55:11,412 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/9b25c1afba7c49f3a68633202362cd48 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/9b25c1afba7c49f3a68633202362cd48 2023-04-09 09:55:11,412 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/2fc52d3e940c430e829135e1b811783f to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/2fc52d3e940c430e829135e1b811783f 2023-04-09 09:55:11,413 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/d5e7a4014a3d45f18d311c5096db71ea to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/d5e7a4014a3d45f18d311c5096db71ea 2023-04-09 09:55:11,414 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/bdee1af652c040af8461c11efba78d10 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/bdee1af652c040af8461c11efba78d10 2023-04-09 09:55:11,414 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/8e021e2fe2834b1e93ab5608740ca248 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/8e021e2fe2834b1e93ab5608740ca248 2023-04-09 09:55:11,415 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/1bb06a59bcfc47e0bdb4b891ddcfbbfb to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/1bb06a59bcfc47e0bdb4b891ddcfbbfb 2023-04-09 09:55:11,416 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/2ae74df5615a47bb894c62dcdc3184a5 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/2ae74df5615a47bb894c62dcdc3184a5 2023-04-09 09:55:11,417 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/299a0beab0a7495da18fd0be015886bc to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/299a0beab0a7495da18fd0be015886bc 2023-04-09 09:55:11,418 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/08525928d962485fb1f7c4ab5ee2b0ef to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/08525928d962485fb1f7c4ab5ee2b0ef 2023-04-09 09:55:11,418 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/6504dc6989ad44a08fd43e651f1835ad to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/6504dc6989ad44a08fd43e651f1835ad 2023-04-09 09:55:11,419 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/bf33f8604b2b42cbb842fc65f57d08b6 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/bf33f8604b2b42cbb842fc65f57d08b6 2023-04-09 09:55:11,420 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/2fbf00d2147f4a77ad7ae3b00c874a03 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/2fbf00d2147f4a77ad7ae3b00c874a03 2023-04-09 09:55:11,421 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/9b0f65f96cbd4d00b92a5cd600b5164a to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/9b0f65f96cbd4d00b92a5cd600b5164a 2023-04-09 09:55:11,422 DEBUG [StoreCloser-TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/c666d36c97e740509374fc9a1624148b to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/c666d36c97e740509374fc9a1624148b 2023-04-09 09:55:11,425 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/recovered.edits/672.seqid, newMaxSeqId=672, maxSeqId=1 2023-04-09 09:55:11,426 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b. 2023-04-09 09:55:11,426 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 026e485e431b415e1bb021c3d14de88b: 2023-04-09 09:55:11,427 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 026e485e431b415e1bb021c3d14de88b 2023-04-09 09:55:11,428 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=026e485e431b415e1bb021c3d14de88b, regionState=CLOSED 2023-04-09 09:55:11,428 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681034111427"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681034111427"}]},"ts":"1681034111427"} 2023-04-09 09:55:11,429 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=32 2023-04-09 09:55:11,429 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=32, state=SUCCESS; CloseRegionProcedure 026e485e431b415e1bb021c3d14de88b, server=jenkins-hbase4.apache.org,42467,1681034019544 in 1.5130 sec 2023-04-09 09:55:11,430 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=31 2023-04-09 09:55:11,431 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=31, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=026e485e431b415e1bb021c3d14de88b, UNASSIGN in 1.5160 sec 2023-04-09 09:55:11,431 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681034111431"}]},"ts":"1681034111431"} 2023-04-09 09:55:11,432 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-09 09:55:11,434 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-09 09:55:11,435 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=31, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5270 sec 2023-04-09 09:55:12,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-09 09:55:12,014 INFO [Listener at localhost/44563] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 31 completed 2023-04-09 09:55:12,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-09 09:55:12,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure2.ProcedureExecutor(1029): Stored pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-09 09:55:12,017 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-09 09:55:12,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-09 09:55:12,017 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=34, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-09 09:55:12,021 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b 2023-04-09 09:55:12,023 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A, FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B, FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C, FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/recovered.edits] 2023-04-09 09:55:12,025 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/3ec276dd67d7402e820a9841325f0fe5 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/3ec276dd67d7402e820a9841325f0fe5 2023-04-09 09:55:12,026 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/cd9fd046bc7747de898c93f2ed40819c to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/cd9fd046bc7747de898c93f2ed40819c 2023-04-09 09:55:12,027 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/f6d7972eb037448d99e0a0ffbabf7a0e to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/A/f6d7972eb037448d99e0a0ffbabf7a0e 2023-04-09 09:55:12,029 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/22af89e1eb16405b8966fa5c82f3df94 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/22af89e1eb16405b8966fa5c82f3df94 2023-04-09 09:55:12,029 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/80e7dcbecdbd4a61840173a2aefe61c5 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/80e7dcbecdbd4a61840173a2aefe61c5 2023-04-09 09:55:12,030 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/c4a77b97982d4a0899d49ea4d573d9ec to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/B/c4a77b97982d4a0899d49ea4d573d9ec 2023-04-09 09:55:12,032 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/b28f29cede974017a86d114b27817917 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/b28f29cede974017a86d114b27817917 2023-04-09 09:55:12,033 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/f8fc755cc0844c60b88e5e93748b6d15 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/f8fc755cc0844c60b88e5e93748b6d15 2023-04-09 09:55:12,034 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/fcdb8c4f0a37490e99cf108d19fc145b to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/C/fcdb8c4f0a37490e99cf108d19fc145b 2023-04-09 09:55:12,037 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/recovered.edits/672.seqid to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b/recovered.edits/672.seqid 2023-04-09 09:55:12,037 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/026e485e431b415e1bb021c3d14de88b 2023-04-09 09:55:12,037 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-09 09:55:12,039 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=34, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-09 09:55:12,040 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-09 09:55:12,042 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-09 09:55:12,043 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=34, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-09 09:55:12,043 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-09 09:55:12,043 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681034112043"}]},"ts":"9223372036854775807"} 2023-04-09 09:55:12,044 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-09 09:55:12,044 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 026e485e431b415e1bb021c3d14de88b, NAME => 'TestAcidGuarantees,,1681034078790.026e485e431b415e1bb021c3d14de88b.', STARTKEY => '', ENDKEY => ''}] 2023-04-09 09:55:12,044 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-09 09:55:12,044 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681034112044"}]},"ts":"9223372036854775807"} 2023-04-09 09:55:12,045 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-09 09:55:12,047 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=34, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-09 09:55:12,048 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=34, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 32 msec 2023-04-09 09:55:12,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-09 09:55:12,118 INFO [Listener at localhost/44563] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 34 completed 2023-04-09 09:55:12,130 INFO [Listener at localhost/44563] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=224 (was 234), OpenFileDescriptor=528 (was 559), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=397 (was 433), ProcessCount=169 (was 169), AvailableMemoryMB=1499 (was 1642) 2023-04-09 09:55:12,140 INFO [Listener at localhost/44563] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=224, OpenFileDescriptor=528, MaxFileDescriptor=60000, SystemLoadAverage=397, ProcessCount=169, AvailableMemoryMB=1499 2023-04-09 09:55:12,142 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-09 09:55:12,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-09 09:55:12,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-09 09:55:12,145 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-09 09:55:12,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 35 2023-04-09 09:55:12,146 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-09 09:55:12,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-09 09:55:12,147 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9 2023-04-09 09:55:12,147 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9 empty. 2023-04-09 09:55:12,148 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9 2023-04-09 09:55:12,148 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-09 09:55:12,154 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-09 09:55:12,155 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => bf27154b49f0831beaa90741ce575bc9, NAME => 'TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp 2023-04-09 09:55:12,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-09 09:55:12,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-09 09:55:12,562 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-09 09:55:12,562 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing bf27154b49f0831beaa90741ce575bc9, disabling compactions & flushes 2023-04-09 09:55:12,562 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:12,562 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:12,562 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. after waiting 0 ms 2023-04-09 09:55:12,562 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:12,562 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:12,562 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for bf27154b49f0831beaa90741ce575bc9: 2023-04-09 09:55:12,564 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-09 09:55:12,565 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681034112565"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681034112565"}]},"ts":"1681034112565"} 2023-04-09 09:55:12,566 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-09 09:55:12,567 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-09 09:55:12,567 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681034112567"}]},"ts":"1681034112567"} 2023-04-09 09:55:12,568 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-09 09:55:12,571 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=bf27154b49f0831beaa90741ce575bc9, ASSIGN}] 2023-04-09 09:55:12,573 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=bf27154b49f0831beaa90741ce575bc9, ASSIGN 2023-04-09 09:55:12,573 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=bf27154b49f0831beaa90741ce575bc9, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,42467,1681034019544; forceNewPlan=false, retain=false 2023-04-09 09:55:12,724 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=bf27154b49f0831beaa90741ce575bc9, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:12,725 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681034112724"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681034112724"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681034112724"}]},"ts":"1681034112724"} 2023-04-09 09:55:12,726 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE; OpenRegionProcedure bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544}] 2023-04-09 09:55:12,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-09 09:55:12,880 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:12,881 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => bf27154b49f0831beaa90741ce575bc9, NAME => 'TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.', STARTKEY => '', ENDKEY => ''} 2023-04-09 09:55:12,881 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees bf27154b49f0831beaa90741ce575bc9 2023-04-09 09:55:12,881 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-09 09:55:12,881 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for bf27154b49f0831beaa90741ce575bc9 2023-04-09 09:55:12,881 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for bf27154b49f0831beaa90741ce575bc9 2023-04-09 09:55:12,882 INFO [StoreOpener-bf27154b49f0831beaa90741ce575bc9-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region bf27154b49f0831beaa90741ce575bc9 2023-04-09 09:55:12,883 DEBUG [StoreOpener-bf27154b49f0831beaa90741ce575bc9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A 2023-04-09 09:55:12,883 DEBUG [StoreOpener-bf27154b49f0831beaa90741ce575bc9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A 2023-04-09 09:55:12,884 INFO [StoreOpener-bf27154b49f0831beaa90741ce575bc9-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-09 09:55:12,884 INFO [StoreOpener-bf27154b49f0831beaa90741ce575bc9-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region bf27154b49f0831beaa90741ce575bc9 columnFamilyName A 2023-04-09 09:55:12,884 INFO [StoreOpener-bf27154b49f0831beaa90741ce575bc9-1] regionserver.HStore(310): Store=bf27154b49f0831beaa90741ce575bc9/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-09 09:55:12,885 INFO [StoreOpener-bf27154b49f0831beaa90741ce575bc9-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region bf27154b49f0831beaa90741ce575bc9 2023-04-09 09:55:12,886 DEBUG [StoreOpener-bf27154b49f0831beaa90741ce575bc9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B 2023-04-09 09:55:12,886 DEBUG [StoreOpener-bf27154b49f0831beaa90741ce575bc9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B 2023-04-09 09:55:12,886 INFO [StoreOpener-bf27154b49f0831beaa90741ce575bc9-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-09 09:55:12,886 INFO [StoreOpener-bf27154b49f0831beaa90741ce575bc9-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region bf27154b49f0831beaa90741ce575bc9 columnFamilyName B 2023-04-09 09:55:12,886 INFO [StoreOpener-bf27154b49f0831beaa90741ce575bc9-1] regionserver.HStore(310): Store=bf27154b49f0831beaa90741ce575bc9/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-09 09:55:12,887 INFO [StoreOpener-bf27154b49f0831beaa90741ce575bc9-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region bf27154b49f0831beaa90741ce575bc9 2023-04-09 09:55:12,887 DEBUG [StoreOpener-bf27154b49f0831beaa90741ce575bc9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C 2023-04-09 09:55:12,888 DEBUG [StoreOpener-bf27154b49f0831beaa90741ce575bc9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C 2023-04-09 09:55:12,888 INFO [StoreOpener-bf27154b49f0831beaa90741ce575bc9-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-09 09:55:12,888 INFO [StoreOpener-bf27154b49f0831beaa90741ce575bc9-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region bf27154b49f0831beaa90741ce575bc9 columnFamilyName C 2023-04-09 09:55:12,888 INFO [StoreOpener-bf27154b49f0831beaa90741ce575bc9-1] regionserver.HStore(310): Store=bf27154b49f0831beaa90741ce575bc9/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-09 09:55:12,888 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:12,889 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9 2023-04-09 09:55:12,889 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9 2023-04-09 09:55:12,890 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-09 09:55:12,891 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for bf27154b49f0831beaa90741ce575bc9 2023-04-09 09:55:12,893 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-09 09:55:12,893 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened bf27154b49f0831beaa90741ce575bc9; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62682810, jitterRate=-0.0659533441066742}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-09 09:55:12,893 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for bf27154b49f0831beaa90741ce575bc9: 2023-04-09 09:55:12,894 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9., pid=37, masterSystemTime=1681034112877 2023-04-09 09:55:12,895 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:12,895 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:12,895 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=bf27154b49f0831beaa90741ce575bc9, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:12,895 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681034112895"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681034112895"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681034112895"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681034112895"}]},"ts":"1681034112895"} 2023-04-09 09:55:12,897 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=36 2023-04-09 09:55:12,897 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; OpenRegionProcedure bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 in 170 msec 2023-04-09 09:55:12,898 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=36, resume processing ppid=35 2023-04-09 09:55:12,898 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=bf27154b49f0831beaa90741ce575bc9, ASSIGN in 326 msec 2023-04-09 09:55:12,899 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-09 09:55:12,899 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681034112899"}]},"ts":"1681034112899"} 2023-04-09 09:55:12,900 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-09 09:55:12,901 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-09 09:55:12,902 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 758 msec 2023-04-09 09:55:13,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-09 09:55:13,250 INFO [Listener at localhost/44563] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 35 completed 2023-04-09 09:55:13,250 DEBUG [Listener at localhost/44563] zookeeper.ReadOnlyZKClient(139): Connect 0x1b77d6d7 to 127.0.0.1:57816 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-09 09:55:13,255 DEBUG [Listener at localhost/44563] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4473f45c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-09 09:55:13,256 DEBUG [Listener at localhost/44563] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-09 09:55:13,258 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35194, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-09 09:55:13,259 DEBUG [Listener at localhost/44563] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-09 09:55:13,260 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38964, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-09 09:55:13,262 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-09 09:55:13,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-09 09:55:13,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure2.ProcedureExecutor(1029): Stored pid=38, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-09 09:55:13,671 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-09 09:55:13,672 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-09 09:55:13,673 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-09 09:55:13,674 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=39, ppid=38, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-09 09:55:13,677 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=bf27154b49f0831beaa90741ce575bc9, REOPEN/MOVE}] 2023-04-09 09:55:13,678 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=bf27154b49f0831beaa90741ce575bc9, REOPEN/MOVE 2023-04-09 09:55:13,678 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=bf27154b49f0831beaa90741ce575bc9, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:13,678 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681034113678"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681034113678"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681034113678"}]},"ts":"1681034113678"} 2023-04-09 09:55:13,679 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=41, ppid=40, state=RUNNABLE; CloseRegionProcedure bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544}] 2023-04-09 09:55:13,831 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close bf27154b49f0831beaa90741ce575bc9 2023-04-09 09:55:13,832 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing bf27154b49f0831beaa90741ce575bc9, disabling compactions & flushes 2023-04-09 09:55:13,832 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:13,832 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:13,832 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. after waiting 0 ms 2023-04-09 09:55:13,832 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:13,835 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-09 09:55:13,836 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:13,836 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for bf27154b49f0831beaa90741ce575bc9: 2023-04-09 09:55:13,836 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: bf27154b49f0831beaa90741ce575bc9 to self. 2023-04-09 09:55:13,837 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed bf27154b49f0831beaa90741ce575bc9 2023-04-09 09:55:13,837 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=bf27154b49f0831beaa90741ce575bc9, regionState=CLOSED 2023-04-09 09:55:13,837 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681034113837"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681034113837"}]},"ts":"1681034113837"} 2023-04-09 09:55:13,839 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=41, resume processing ppid=40 2023-04-09 09:55:13,839 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=40, state=SUCCESS; CloseRegionProcedure bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 in 159 msec 2023-04-09 09:55:13,840 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=bf27154b49f0831beaa90741ce575bc9, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,42467,1681034019544; forceNewPlan=false, retain=true 2023-04-09 09:55:13,990 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=bf27154b49f0831beaa90741ce575bc9, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:13,990 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681034113990"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681034113990"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681034113990"}]},"ts":"1681034113990"} 2023-04-09 09:55:13,992 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=42, ppid=40, state=RUNNABLE; OpenRegionProcedure bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544}] 2023-04-09 09:55:14,146 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:14,146 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => bf27154b49f0831beaa90741ce575bc9, NAME => 'TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.', STARTKEY => '', ENDKEY => ''} 2023-04-09 09:55:14,147 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees bf27154b49f0831beaa90741ce575bc9 2023-04-09 09:55:14,147 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-09 09:55:14,147 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for bf27154b49f0831beaa90741ce575bc9 2023-04-09 09:55:14,147 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for bf27154b49f0831beaa90741ce575bc9 2023-04-09 09:55:14,148 INFO [StoreOpener-bf27154b49f0831beaa90741ce575bc9-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region bf27154b49f0831beaa90741ce575bc9 2023-04-09 09:55:14,149 DEBUG [StoreOpener-bf27154b49f0831beaa90741ce575bc9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A 2023-04-09 09:55:14,149 DEBUG [StoreOpener-bf27154b49f0831beaa90741ce575bc9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A 2023-04-09 09:55:14,150 INFO [StoreOpener-bf27154b49f0831beaa90741ce575bc9-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-09 09:55:14,150 INFO [StoreOpener-bf27154b49f0831beaa90741ce575bc9-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region bf27154b49f0831beaa90741ce575bc9 columnFamilyName A 2023-04-09 09:55:14,151 INFO [StoreOpener-bf27154b49f0831beaa90741ce575bc9-1] regionserver.HStore(310): Store=bf27154b49f0831beaa90741ce575bc9/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-09 09:55:14,151 INFO [StoreOpener-bf27154b49f0831beaa90741ce575bc9-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region bf27154b49f0831beaa90741ce575bc9 2023-04-09 09:55:14,152 DEBUG [StoreOpener-bf27154b49f0831beaa90741ce575bc9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B 2023-04-09 09:55:14,152 DEBUG [StoreOpener-bf27154b49f0831beaa90741ce575bc9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B 2023-04-09 09:55:14,152 INFO [StoreOpener-bf27154b49f0831beaa90741ce575bc9-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-09 09:55:14,153 INFO [StoreOpener-bf27154b49f0831beaa90741ce575bc9-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region bf27154b49f0831beaa90741ce575bc9 columnFamilyName B 2023-04-09 09:55:14,153 INFO [StoreOpener-bf27154b49f0831beaa90741ce575bc9-1] regionserver.HStore(310): Store=bf27154b49f0831beaa90741ce575bc9/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-09 09:55:14,153 INFO [StoreOpener-bf27154b49f0831beaa90741ce575bc9-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region bf27154b49f0831beaa90741ce575bc9 2023-04-09 09:55:14,154 DEBUG [StoreOpener-bf27154b49f0831beaa90741ce575bc9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C 2023-04-09 09:55:14,154 DEBUG [StoreOpener-bf27154b49f0831beaa90741ce575bc9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C 2023-04-09 09:55:14,154 INFO [StoreOpener-bf27154b49f0831beaa90741ce575bc9-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-09 09:55:14,154 INFO [StoreOpener-bf27154b49f0831beaa90741ce575bc9-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region bf27154b49f0831beaa90741ce575bc9 columnFamilyName C 2023-04-09 09:55:14,154 INFO [StoreOpener-bf27154b49f0831beaa90741ce575bc9-1] regionserver.HStore(310): Store=bf27154b49f0831beaa90741ce575bc9/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-09 09:55:14,155 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:14,155 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9 2023-04-09 09:55:14,156 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9 2023-04-09 09:55:14,157 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-09 09:55:14,158 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for bf27154b49f0831beaa90741ce575bc9 2023-04-09 09:55:14,159 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened bf27154b49f0831beaa90741ce575bc9; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73618385, jitterRate=0.09699942171573639}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-09 09:55:14,159 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for bf27154b49f0831beaa90741ce575bc9: 2023-04-09 09:55:14,159 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9., pid=42, masterSystemTime=1681034114143 2023-04-09 09:55:14,160 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:14,160 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:14,161 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=bf27154b49f0831beaa90741ce575bc9, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,161 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681034114161"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681034114161"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681034114161"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681034114161"}]},"ts":"1681034114161"} 2023-04-09 09:55:14,163 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=42, resume processing ppid=40 2023-04-09 09:55:14,163 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=40, state=SUCCESS; OpenRegionProcedure bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 in 170 msec 2023-04-09 09:55:14,164 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=40, resume processing ppid=39 2023-04-09 09:55:14,164 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=39, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=bf27154b49f0831beaa90741ce575bc9, REOPEN/MOVE in 486 msec 2023-04-09 09:55:14,166 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=38 2023-04-09 09:55:14,166 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=38, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 491 msec 2023-04-09 09:55:14,168 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=38, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 905 msec 2023-04-09 09:55:14,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=38 2023-04-09 09:55:14,169 DEBUG [Listener at localhost/44563] zookeeper.ReadOnlyZKClient(139): Connect 0x6434ce29 to 127.0.0.1:57816 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-09 09:55:14,174 DEBUG [Listener at localhost/44563] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@41507d16, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-09 09:55:14,174 DEBUG [Listener at localhost/44563] zookeeper.ReadOnlyZKClient(139): Connect 0x0bb27fd4 to 127.0.0.1:57816 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-09 09:55:14,179 DEBUG [Listener at localhost/44563] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@49b5bd0d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-09 09:55:14,179 DEBUG [Listener at localhost/44563] zookeeper.ReadOnlyZKClient(139): Connect 0x445919fd to 127.0.0.1:57816 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-09 09:55:14,186 DEBUG [Listener at localhost/44563] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5c8d927d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-09 09:55:14,187 DEBUG [Listener at localhost/44563] zookeeper.ReadOnlyZKClient(139): Connect 0x4d010713 to 127.0.0.1:57816 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-09 09:55:14,190 DEBUG [Listener at localhost/44563] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@419b69bf, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-09 09:55:14,191 DEBUG [Listener at localhost/44563] zookeeper.ReadOnlyZKClient(139): Connect 0x69100738 to 127.0.0.1:57816 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-09 09:55:14,194 DEBUG [Listener at localhost/44563] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2c6cac96, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-09 09:55:14,195 DEBUG [Listener at localhost/44563] zookeeper.ReadOnlyZKClient(139): Connect 0x06a0df34 to 127.0.0.1:57816 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-09 09:55:14,200 DEBUG [Listener at localhost/44563] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4fe1db6a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-09 09:55:14,200 DEBUG [Listener at localhost/44563] zookeeper.ReadOnlyZKClient(139): Connect 0x10ee5f06 to 127.0.0.1:57816 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-09 09:55:14,207 DEBUG [Listener at localhost/44563] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@10377b50, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-09 09:55:14,207 DEBUG [Listener at localhost/44563] zookeeper.ReadOnlyZKClient(139): Connect 0x21128196 to 127.0.0.1:57816 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-09 09:55:14,216 DEBUG [Listener at localhost/44563] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4366fcdd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-09 09:55:14,217 DEBUG [Listener at localhost/44563] zookeeper.ReadOnlyZKClient(139): Connect 0x5f9e86d9 to 127.0.0.1:57816 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-09 09:55:14,221 DEBUG [Listener at localhost/44563] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6a35a28d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-09 09:55:14,221 DEBUG [Listener at localhost/44563] zookeeper.ReadOnlyZKClient(139): Connect 0x711ebaf2 to 127.0.0.1:57816 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-09 09:55:14,227 DEBUG [Listener at localhost/44563] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5a1d4246, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-09 09:55:14,238 DEBUG [hconnection-0x51e12744-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-09 09:55:14,239 DEBUG [hconnection-0x5eacec96-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-09 09:55:14,239 DEBUG [hconnection-0x6454c880-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-09 09:55:14,240 DEBUG [hconnection-0x64821d66-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-09 09:55:14,240 DEBUG [hconnection-0x79927ed3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-09 09:55:14,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:14,240 DEBUG [hconnection-0x15165fbc-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-09 09:55:14,241 DEBUG [hconnection-0x3cb9f896-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-09 09:55:14,241 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:32772, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-09 09:55:14,241 DEBUG [hconnection-0x4feac512-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-09 09:55:14,241 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:32810, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-09 09:55:14,241 DEBUG [hconnection-0x484e08a8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-09 09:55:14,241 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:32856, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-09 09:55:14,243 DEBUG [hconnection-0x23680d45-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-09 09:55:14,246 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:32798, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-09 09:55:14,246 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:32788, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-09 09:55:14,246 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:32840, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-09 09:55:14,246 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:32826, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-09 09:55:14,247 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:32870, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-09 09:55:14,247 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:32866, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-09 09:55:14,247 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:32882, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-09 09:55:14,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:14,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:14,250 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:14,250 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:14,250 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:14,250 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:14,250 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:14,250 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:14,252 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:14,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:14,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:14,252 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:14,252 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,253 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:14,253 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:14,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:14,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:14,253 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:14,263 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:14,263 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:14,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on bf27154b49f0831beaa90741ce575bc9 2023-04-09 09:55:14,263 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:14,264 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:14,264 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bf27154b49f0831beaa90741ce575bc9 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-09 09:55:14,264 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:14,264 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:14,264 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:14,264 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf27154b49f0831beaa90741ce575bc9, store=A 2023-04-09 09:55:14,264 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. started... 2023-04-09 09:55:14,264 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:14,264 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. as already flushing 2023-04-09 09:55:14,264 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf27154b49f0831beaa90741ce575bc9, store=B 2023-04-09 09:55:14,264 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:14,264 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:14,264 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:14,264 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:14,264 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf27154b49f0831beaa90741ce575bc9, store=C 2023-04-09 09:55:14,264 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,265 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:14,265 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:14,265 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:14,266 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:14,266 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:14,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:14,267 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:14,267 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:14,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:14,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:14,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:14,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:14,268 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:14,268 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63b8011a[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:14,268 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:14,268 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:14,270 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:14,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:14,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:14,271 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:14,271 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,271 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:14,271 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:14,271 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:14,274 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:14,274 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:14,275 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,275 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:14,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:14,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:14,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:14,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:14,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:14,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:14,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:14,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:14,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:14,277 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:14,277 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:14,277 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:14,277 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:14,277 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:14,279 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:14,279 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:14,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:14,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:14,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:14,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:14,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:14,279 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:14,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:14,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:14,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:14,280 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:14,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:14,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:14,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:14,282 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:14,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:14,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,285 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,285 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:14,286 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:14,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:14,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:14,286 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:14,286 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:14,286 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:14,286 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:14,286 DEBUG [Thread-2431] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:14,286 DEBUG [Thread-2431] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:14,285 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:14,286 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:14,286 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,286 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:14,286 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:14,286 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:14,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:14,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:14,295 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:14,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32856 deadline: 1681034174290, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,295 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:14,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 4 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32788 deadline: 1681034174290, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,295 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034114279, max timestamp=1681034114294Num uniques 859; 2023-04-09 09:55:14,295 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:14,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 5 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32882 deadline: 1681034174290, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,295 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:55:14,296 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034114279, max timestamp=1681034114294Num uniques 859; 2023-04-09 09:55:14,296 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:55:14,296 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034114279, max timestamp=1681034114294Num uniques 859; 2023-04-09 09:55:14,296 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:55:14,312 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:14,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32870 deadline: 1681034174308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,312 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:14,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32772 deadline: 1681034174308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,386 DEBUG [Thread-2431] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:14,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:14,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:14,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:14,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:14,393 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:14,393 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:14,393 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:14,393 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:14,393 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:14,393 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:14,396 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:14,396 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:14,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:14,396 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,396 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:14,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:14,397 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:14,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:14,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:14,397 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:14,399 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:14,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32856 deadline: 1681034174396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,399 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:14,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32788 deadline: 1681034174396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,399 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:14,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32882 deadline: 1681034174396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,400 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:14,400 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:14,400 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:14,400 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:14,400 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:14,404 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:14,404 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:14,404 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:14,404 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. started... 2023-04-09 09:55:14,404 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. as already flushing 2023-04-09 09:55:14,404 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:14,404 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:14,404 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:14,404 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,404 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:14,404 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:14,406 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,406 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:14,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:14,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:14,406 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:14,406 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:14,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:14,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:14,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:14,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:14,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:14,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5161f298[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:14,407 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:14,407 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:14,408 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:14,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:14,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:14,409 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:14,409 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,409 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:14,409 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:14,409 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:14,410 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,410 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:14,410 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:14,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:14,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:14,410 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:14,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:14,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:14,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:14,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:14,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:14,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:14,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:14,412 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:14,412 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:14,412 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:14,412 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:14,412 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:14,413 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:14,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32772 deadline: 1681034174413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,414 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:14,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32870 deadline: 1681034174414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,415 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:14,415 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:14,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:14,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:14,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:14,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:14,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:14,415 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:14,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:14,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:14,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:14,415 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:14,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:14,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:14,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:14,416 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:14,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:14,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,418 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:14,418 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:14,419 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,419 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:14,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:14,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:14,418 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:14,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:14,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:14,419 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:14,419 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:14,419 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,419 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:14,419 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:14,419 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:14,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:14,419 DEBUG [Thread-2431] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:14,419 DEBUG [Thread-2431] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:14,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:14,519 DEBUG [Thread-2431] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:14,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:14,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:14,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:14,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:14,523 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:14,523 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:14,523 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:14,523 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:14,523 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:14,524 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:14,525 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,525 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:14,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:14,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:14,525 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:14,525 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:14,526 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:14,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:14,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:14,526 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:14,526 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:14,526 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:14,526 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:14,526 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:14,526 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:14,528 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:14,528 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:14,528 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:14,528 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. started... 2023-04-09 09:55:14,528 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. as already flushing 2023-04-09 09:55:14,528 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:14,528 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:14,528 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:14,528 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,528 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:14,528 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:14,530 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,530 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:14,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:14,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:14,530 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:14,530 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:14,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:14,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:14,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:14,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:14,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:14,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38e15756[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:14,531 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:14,531 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:14,533 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:14,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:14,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:14,533 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:14,533 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:14,533 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,533 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:14,533 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:14,534 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:14,534 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:14,534 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,535 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:14,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:14,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:14,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:14,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:14,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:14,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:14,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:14,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:14,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:14,537 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:14,537 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:14,537 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:14,537 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:14,537 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:14,538 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:14,538 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:14,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:14,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:14,538 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:14,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:14,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:14,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:14,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:14,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:14,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:14,539 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:14,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:14,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:14,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:14,541 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:14,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:14,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,543 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,543 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:14,543 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:14,543 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:14,544 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:14,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:14,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:14,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:14,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:14,544 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:14,544 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:14,543 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,544 DEBUG [Thread-2431] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:14,544 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:14,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:14,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:14,544 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:14,544 DEBUG [Thread-2431] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:14,544 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:14,603 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:14,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32856 deadline: 1681034174600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,603 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:14,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32882 deadline: 1681034174600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,603 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:14,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32788 deadline: 1681034174601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,617 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:14,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32772 deadline: 1681034174615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,617 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:14,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32870 deadline: 1681034174616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,644 DEBUG [Thread-2431] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:14,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:14,646 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:14,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:14,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:14,647 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:14,648 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:14,648 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:14,648 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:14,648 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:14,648 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:14,652 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:14,652 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:14,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:14,653 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,653 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:14,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:14,653 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:14,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:14,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:14,654 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:14,654 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:14,654 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:14,654 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:14,654 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:14,654 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:14,654 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:14,654 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:14,654 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:14,654 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. started... 2023-04-09 09:55:14,654 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. as already flushing 2023-04-09 09:55:14,654 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:14,654 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:14,654 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:14,654 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,655 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:14,655 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:14,656 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,656 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:14,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:14,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:14,656 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:14,656 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:14,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:14,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:14,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:14,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:14,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:14,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@117577f6[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:14,658 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:14,658 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:14,659 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:14,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:14,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:14,659 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:14,659 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:14,659 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:14,659 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,659 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:14,660 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,660 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:14,660 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:14,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:14,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:14,660 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:14,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:14,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:14,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:14,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:14,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:14,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:14,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:14,662 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:14,662 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:14,662 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:14,663 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:14,663 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:14,665 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:14,665 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:14,665 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:14,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:14,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:14,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:14,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:14,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:14,666 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:14,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:14,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:14,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:14,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:14,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:14,667 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:14,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:14,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:14,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,669 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,669 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:14,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:14,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:14,669 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:14,670 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:14,669 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:14,670 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:14,670 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,670 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:14,670 DEBUG [Thread-2431] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:14,670 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:14,670 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:14,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:14,670 DEBUG [Thread-2431] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:14,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:14,671 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:14,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:14,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:14,691 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304092ed9da273baa44c78f1ff213e9868500 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304092ed9da273baa44c78f1ff213e9868500 2023-04-09 09:55:14,696 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/A/6414cc9aef4a4bb18f9a7cc6cc04fd59 2023-04-09 09:55:14,770 DEBUG [Thread-2431] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:14,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:14,772 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:14,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:14,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:14,773 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:14,773 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:14,773 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:14,774 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:14,774 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:14,774 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:14,775 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:14,775 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:14,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:14,775 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,775 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:14,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:14,776 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:14,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:14,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:14,776 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:14,776 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:14,776 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:14,776 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:14,777 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:14,777 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:14,777 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:14,777 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:14,777 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:14,777 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. started... 2023-04-09 09:55:14,777 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. as already flushing 2023-04-09 09:55:14,777 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:14,777 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:14,777 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:14,777 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,777 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:14,777 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:14,779 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,779 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:14,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:14,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:14,779 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:14,779 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:14,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:14,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:14,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:14,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:14,780 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:14,780 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32e60413[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:14,780 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:14,780 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:14,783 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:14,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:14,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:14,783 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:14,783 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:14,783 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:14,783 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,783 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:14,784 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,784 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:14,785 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:14,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:14,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:14,785 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:14,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:14,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:14,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:14,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:14,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:14,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:14,787 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:14,787 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:14,787 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:14,787 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:14,787 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:14,787 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:14,788 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:14,788 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:14,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:14,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:14,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:14,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:14,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:14,788 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:14,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:14,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:14,789 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:14,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:14,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:14,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:14,790 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:14,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:14,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:14,792 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:14,792 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,792 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:14,792 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:14,792 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:14,792 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,792 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:14,792 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:14,792 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:14,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:14,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:14,792 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:14,792 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:14,792 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:14,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:14,794 DEBUG [Thread-2431] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:14,794 DEBUG [Thread-2431] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:14,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:14,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:14,894 DEBUG [Thread-2431] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:14,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:14,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:14,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:14,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:14,897 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:14,897 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:14,897 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:14,897 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:14,897 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:14,898 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:14,900 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:14,900 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:14,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:14,900 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,900 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:14,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:14,900 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:14,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:14,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:14,901 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:14,901 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:14,901 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:14,901 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:14,901 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:14,901 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:14,901 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:14,901 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:14,901 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:14,901 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. started... 2023-04-09 09:55:14,901 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. as already flushing 2023-04-09 09:55:14,902 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:14,902 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:14,902 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:14,902 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,902 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:14,902 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:14,903 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,903 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:14,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:14,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:14,903 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:14,903 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:14,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:14,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:14,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:14,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:14,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:14,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c4dd5e5[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:14,905 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:14,905 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:14,905 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:14,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32882 deadline: 1681034174904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,906 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:14,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:14,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:14,906 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,907 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:14,906 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:14,907 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:14,907 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:14,907 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:14,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32788 deadline: 1681034174904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,908 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:14,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32856 deadline: 1681034174905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,908 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,908 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:14,908 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:14,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:14,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:14,908 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:14,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:14,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:14,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:14,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:14,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:14,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:14,910 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:14,910 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:14,910 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:14,910 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:14,911 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:14,911 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:14,913 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:14,913 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:14,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:14,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:14,913 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:14,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:14,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:14,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:14,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:14,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:14,913 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:14,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:14,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:14,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:14,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:14,914 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:14,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:14,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,917 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:14,917 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:14,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:14,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:14,917 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,917 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:14,917 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:14,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:14,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:14,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:14,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:14,917 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,918 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:14,918 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:14,918 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:14,917 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:14,918 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:14,918 DEBUG [Thread-2431] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:14,918 DEBUG [Thread-2431] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:14,920 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:14,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32772 deadline: 1681034174919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:14,920 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:14,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32870 deadline: 1681034174919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:15,018 DEBUG [Thread-2431] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:15,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:15,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:15,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:15,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:15,021 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:15,022 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:15,022 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:15,022 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:15,022 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:15,022 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:15,023 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:15,023 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:15,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:15,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:15,024 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:15,024 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:15,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:15,024 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:15,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:15,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:15,024 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:15,025 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:15,025 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:15,025 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:15,025 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:15,025 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:15,025 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:15,025 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:15,025 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:15,025 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. started... 2023-04-09 09:55:15,025 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. as already flushing 2023-04-09 09:55:15,025 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:15,025 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:15,025 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:15,025 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:15,025 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:15,025 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:15,027 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:15,027 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:15,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:15,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:15,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:15,027 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:15,027 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:15,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:15,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:15,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:15,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:15,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:15,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:15,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ea474dd[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:15,029 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:15,029 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:15,034 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:15,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:15,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:15,034 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:15,034 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:15,034 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:15,034 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:15,034 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:15,035 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:15,035 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:15,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:15,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:15,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:15,035 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:15,036 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:15,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:15,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:15,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:15,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:15,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:15,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:15,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:15,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:15,038 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:15,038 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:15,038 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:15,038 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:15,038 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:15,038 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:15,039 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:15,039 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:15,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:15,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:15,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:15,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:15,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:15,039 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:15,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:15,040 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:15,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:15,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:15,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:15,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:15,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:15,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:15,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:15,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:15,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:15,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:15,042 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:15,044 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:15,044 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:15,044 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:15,044 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:15,044 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:15,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:15,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:15,044 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:15,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:15,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:15,044 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:15,044 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:15,044 DEBUG [Thread-2431] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:15,044 DEBUG [Thread-2431] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:15,044 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:15,044 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:15,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:15,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:15,107 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/B/0f73b8ad946d4199b91fddf0bc874460 2023-04-09 09:55:15,118 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/C/65ccd0efd6894cbc93de631d4f2b993f 2023-04-09 09:55:15,121 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/A/6414cc9aef4a4bb18f9a7cc6cc04fd59 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/6414cc9aef4a4bb18f9a7cc6cc04fd59 2023-04-09 09:55:15,125 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/6414cc9aef4a4bb18f9a7cc6cc04fd59, entries=100, sequenceid=15, filesize=18.4 K 2023-04-09 09:55:15,126 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/B/0f73b8ad946d4199b91fddf0bc874460 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/0f73b8ad946d4199b91fddf0bc874460 2023-04-09 09:55:15,129 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/0f73b8ad946d4199b91fddf0bc874460, entries=100, sequenceid=15, filesize=9.3 K 2023-04-09 09:55:15,130 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/C/65ccd0efd6894cbc93de631d4f2b993f as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/65ccd0efd6894cbc93de631d4f2b993f 2023-04-09 09:55:15,133 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/65ccd0efd6894cbc93de631d4f2b993f, entries=100, sequenceid=15, filesize=9.3 K 2023-04-09 09:55:15,134 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for bf27154b49f0831beaa90741ce575bc9 in 871ms, sequenceid=15, compaction requested=false 2023-04-09 09:55:15,134 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-09 09:55:15,134 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bf27154b49f0831beaa90741ce575bc9: 2023-04-09 09:55:15,144 DEBUG [Thread-2431] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:15,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:15,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:15,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:15,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:15,147 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:15,147 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:15,147 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:15,147 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:15,148 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:15,148 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:15,150 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:15,150 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:15,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:15,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:15,150 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:15,150 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:15,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:15,150 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:15,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:15,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:15,151 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:15,151 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:15,151 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:15,151 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:15,151 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:15,151 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:15,151 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:15,151 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:15,151 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:15,151 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. started... 2023-04-09 09:55:15,151 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bf27154b49f0831beaa90741ce575bc9 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-09 09:55:15,152 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf27154b49f0831beaa90741ce575bc9, store=A 2023-04-09 09:55:15,152 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:55:15,152 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf27154b49f0831beaa90741ce575bc9, store=B 2023-04-09 09:55:15,152 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:55:15,152 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf27154b49f0831beaa90741ce575bc9, store=C 2023-04-09 09:55:15,152 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:55:15,164 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230409937ce52a963b4e0da2ca3c25bc286089 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409937ce52a963b4e0da2ca3c25bc286089 2023-04-09 09:55:15,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on bf27154b49f0831beaa90741ce575bc9 2023-04-09 09:55:15,409 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. as already flushing 2023-04-09 09:55:15,421 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:15,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32856 deadline: 1681034175416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:15,422 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:15,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32882 deadline: 1681034175418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:15,423 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:15,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32788 deadline: 1681034175419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:15,424 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:15,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32772 deadline: 1681034175421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:15,429 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:15,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32870 deadline: 1681034175425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:15,526 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:15,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32856 deadline: 1681034175522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:15,528 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:15,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32882 deadline: 1681034175523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:15,528 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:15,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32788 deadline: 1681034175523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:15,569 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=52, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/A/7d245810b3aa4e10a7b61f5caa0e04c8 2023-04-09 09:55:15,735 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:15,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32856 deadline: 1681034175728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:15,735 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:15,735 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:15,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32788 deadline: 1681034175730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:15,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32882 deadline: 1681034175730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:15,979 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=52 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/B/c0aa63628b4548bba9707d3173ed0b3f 2023-04-09 09:55:15,989 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=52 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/C/c3d6fa61659e4dd5a374c671526fbb25 2023-04-09 09:55:15,993 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/A/7d245810b3aa4e10a7b61f5caa0e04c8 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/7d245810b3aa4e10a7b61f5caa0e04c8 2023-04-09 09:55:15,996 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/7d245810b3aa4e10a7b61f5caa0e04c8, entries=150, sequenceid=52, filesize=25.1 K 2023-04-09 09:55:15,997 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/B/c0aa63628b4548bba9707d3173ed0b3f as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/c0aa63628b4548bba9707d3173ed0b3f 2023-04-09 09:55:16,000 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/c0aa63628b4548bba9707d3173ed0b3f, entries=150, sequenceid=52, filesize=11.6 K 2023-04-09 09:55:16,001 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/C/c3d6fa61659e4dd5a374c671526fbb25 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/c3d6fa61659e4dd5a374c671526fbb25 2023-04-09 09:55:16,004 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/c3d6fa61659e4dd5a374c671526fbb25, entries=150, sequenceid=52, filesize=11.6 K 2023-04-09 09:55:16,005 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for bf27154b49f0831beaa90741ce575bc9 in 854ms, sequenceid=52, compaction requested=false 2023-04-09 09:55:16,006 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bf27154b49f0831beaa90741ce575bc9: 2023-04-09 09:55:16,006 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:16,006 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:16,006 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:16,006 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,006 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:16,006 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:16,008 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:16,008 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:16,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:16,008 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:16,008 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:16,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:16,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:16,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:16,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:16,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:16,010 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64547238[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:16,010 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:16,010 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:16,011 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:16,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:16,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:16,011 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:16,011 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:16,011 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:16,011 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,011 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:16,013 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:16,013 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:16,013 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,013 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:16,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:16,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:16,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:16,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:16,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:16,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:16,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:16,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:16,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:16,015 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:16,015 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:16,015 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:16,015 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:16,015 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:16,018 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:16,018 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:16,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:16,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:16,018 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:16,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:16,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:16,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:16,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:16,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:16,018 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:16,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:16,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:16,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:16,019 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:16,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:16,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:16,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,021 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,021 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:16,021 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:16,022 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:16,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:16,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:16,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:16,021 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:16,022 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:16,022 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:16,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:16,022 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,022 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:16,022 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:16,022 DEBUG [Thread-2431] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:16,022 DEBUG [Thread-2431] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:16,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:16,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:16,022 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:16,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on bf27154b49f0831beaa90741ce575bc9 2023-04-09 09:55:16,040 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bf27154b49f0831beaa90741ce575bc9 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-09 09:55:16,040 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf27154b49f0831beaa90741ce575bc9, store=A 2023-04-09 09:55:16,040 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:16,040 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf27154b49f0831beaa90741ce575bc9, store=B 2023-04-09 09:55:16,041 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:16,041 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf27154b49f0831beaa90741ce575bc9, store=C 2023-04-09 09:55:16,041 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:16,076 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:16,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32882 deadline: 1681034176071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,076 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:16,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32788 deadline: 1681034176076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,077 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:16,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034116069, max timestamp=1681034116075Num uniques 859; 2023-04-09 09:55:16,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034116069, max timestamp=1681034116075Num uniques 859; 2023-04-09 09:55:16,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32856 deadline: 1681034176076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034116069, max timestamp=1681034116075Num uniques 859; 2023-04-09 09:55:16,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:55:16,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:55:16,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:55:16,122 DEBUG [Thread-2431] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:16,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:16,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:16,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:16,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:16,126 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:16,126 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:16,126 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:16,126 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:16,126 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:16,126 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:16,127 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,127 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:16,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:16,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:16,128 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,128 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:16,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:16,128 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:16,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:16,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:16,129 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:16,129 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:16,129 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:16,129 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:16,129 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:16,129 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:16,131 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:16,131 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:16,131 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:16,131 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. started... 2023-04-09 09:55:16,131 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. as already flushing 2023-04-09 09:55:16,131 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:16,131 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:16,131 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:16,131 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,131 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:16,131 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:16,132 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:16,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:16,133 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:16,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:16,133 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:16,133 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:16,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:16,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:16,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:16,134 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:16,134 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6610243d[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:16,134 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:16,134 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:16,145 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:16,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:16,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:16,145 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:16,145 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:16,145 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,145 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:16,145 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:16,147 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:16,147 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,147 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:16,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:16,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:16,147 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:16,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:16,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:16,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:16,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:16,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:16,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:16,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:16,149 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:16,149 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:16,149 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:16,150 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:16,150 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:16,151 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:16,151 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:16,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:16,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:16,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:16,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:16,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:16,151 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:16,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:16,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:16,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:16,152 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:16,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:16,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:16,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:16,153 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:16,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:16,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,155 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,155 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:16,155 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:16,155 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:16,155 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:16,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:16,155 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,155 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:16,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:16,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:16,155 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:16,155 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:16,155 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:16,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:16,156 DEBUG [Thread-2431] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:16,156 DEBUG [Thread-2431] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:16,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:16,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:16,211 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:16,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32882 deadline: 1681034176205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,211 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:16,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32788 deadline: 1681034176208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,214 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:16,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32856 deadline: 1681034176211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,256 DEBUG [Thread-2431] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:16,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:16,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:16,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:16,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:16,259 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:16,259 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:16,259 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:16,259 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:16,259 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:16,259 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:16,261 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:16,261 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:16,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:16,262 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,262 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:16,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:16,262 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:16,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:16,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:16,262 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:16,263 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:16,263 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:16,263 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:16,263 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:16,263 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:16,263 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:16,263 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:16,263 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:16,263 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. started... 2023-04-09 09:55:16,263 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. as already flushing 2023-04-09 09:55:16,263 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:16,263 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:16,263 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:16,263 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,263 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:16,263 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:16,265 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,265 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:16,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:16,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:16,265 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:16,265 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:16,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:16,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:16,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:16,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:16,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:16,266 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:16,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@647027e7[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:16,267 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:16,268 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:16,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:16,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:16,268 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:16,268 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:16,268 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,268 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:16,268 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:16,276 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:16,276 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,276 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:16,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:16,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:16,276 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:16,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:16,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:16,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:16,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:16,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:16,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:16,278 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:16,278 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:16,278 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:16,278 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:16,278 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:16,278 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:16,280 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:16,280 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:16,280 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:16,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:16,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:16,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:16,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:16,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:16,280 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:16,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:16,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:16,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:16,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:16,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:16,281 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:16,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:16,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:16,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,284 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,284 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:16,284 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:16,284 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:16,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:16,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:16,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:16,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:16,284 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:16,284 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:16,284 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,284 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:16,284 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:16,284 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:16,284 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:16,284 DEBUG [Thread-2431] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:16,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:16,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:16,284 DEBUG [Thread-2431] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:16,316 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:16,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32882 deadline: 1681034176312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,316 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:16,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32788 deadline: 1681034176312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,317 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:16,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32856 deadline: 1681034176315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,385 DEBUG [Thread-2431] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:16,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:16,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:16,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:16,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:16,388 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:16,388 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:16,388 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:16,388 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:16,389 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:16,389 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:16,390 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:16,390 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:16,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:16,390 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,390 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:16,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:16,391 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:16,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:16,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:16,391 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:16,391 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:16,391 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:16,391 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:16,391 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:16,391 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:16,391 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:16,391 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:16,392 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:16,392 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. started... 2023-04-09 09:55:16,392 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. as already flushing 2023-04-09 09:55:16,392 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:16,392 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:16,392 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:16,392 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,392 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:16,392 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:16,393 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,393 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:16,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:16,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:16,394 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:16,394 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:16,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:16,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:16,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:16,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:16,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:16,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@190fb528[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:16,395 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:16,395 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:16,396 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:16,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:16,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:16,396 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:16,396 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:16,396 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:16,396 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,396 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:16,398 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:16,398 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:16,398 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,399 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:16,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:16,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:16,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:16,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:16,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:16,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:16,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:16,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:16,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:16,401 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:16,401 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:16,401 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:16,401 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:16,401 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:16,402 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:16,403 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:16,403 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:16,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:16,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:16,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:16,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:16,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:16,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:16,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:16,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:16,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:16,403 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:16,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:16,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:16,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,405 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:16,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:16,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,407 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,407 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:16,407 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:16,407 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:16,407 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:16,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:16,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:16,407 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,407 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:16,407 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:16,407 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:16,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:16,407 DEBUG [Thread-2431] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:16,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:16,407 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:16,407 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:16,407 DEBUG [Thread-2431] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:16,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:16,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:16,436 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:16,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32772 deadline: 1681034176431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,443 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:16,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32870 deadline: 1681034176439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,457 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230409e901b38d3a05488bb0f4d2603f57c560 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409e901b38d3a05488bb0f4d2603f57c560 2023-04-09 09:55:16,465 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=63, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/A/27f622741ce54d2c85f56052891edd91 2023-04-09 09:55:16,476 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=63 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/B/d6a39d3d76dd49cbac4b5fe71c4bc827 2023-04-09 09:55:16,485 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=63 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/C/5bd2e8b8bdc34f83a64e17fa582ef1d8 2023-04-09 09:55:16,489 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/A/27f622741ce54d2c85f56052891edd91 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/27f622741ce54d2c85f56052891edd91 2023-04-09 09:55:16,492 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/27f622741ce54d2c85f56052891edd91, entries=200, sequenceid=63, filesize=31.9 K 2023-04-09 09:55:16,493 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/B/d6a39d3d76dd49cbac4b5fe71c4bc827 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/d6a39d3d76dd49cbac4b5fe71c4bc827 2023-04-09 09:55:16,496 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/d6a39d3d76dd49cbac4b5fe71c4bc827, entries=150, sequenceid=63, filesize=11.6 K 2023-04-09 09:55:16,496 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/C/5bd2e8b8bdc34f83a64e17fa582ef1d8 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/5bd2e8b8bdc34f83a64e17fa582ef1d8 2023-04-09 09:55:16,499 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/5bd2e8b8bdc34f83a64e17fa582ef1d8, entries=150, sequenceid=63, filesize=11.6 K 2023-04-09 09:55:16,500 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for bf27154b49f0831beaa90741ce575bc9 in 460ms, sequenceid=63, compaction requested=true 2023-04-09 09:55:16,500 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bf27154b49f0831beaa90741ce575bc9: 2023-04-09 09:55:16,500 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:55:16,500 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-09 09:55:16,500 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:55:16,500 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-09 09:55:16,500 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:55:16,501 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 77230 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-09 09:55:16,501 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1912): bf27154b49f0831beaa90741ce575bc9/A is initiating minor compaction (all files) 2023-04-09 09:55:16,501 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bf27154b49f0831beaa90741ce575bc9/A in TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:16,501 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/6414cc9aef4a4bb18f9a7cc6cc04fd59, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/7d245810b3aa4e10a7b61f5caa0e04c8, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/27f622741ce54d2c85f56052891edd91] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp, totalSize=75.4 K 2023-04-09 09:55:16,502 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-09 09:55:16,502 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 6414cc9aef4a4bb18f9a7cc6cc04fd59, keycount=100, bloomtype=ROW, size=18.4 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1681034114255 2023-04-09 09:55:16,502 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): bf27154b49f0831beaa90741ce575bc9/B is initiating minor compaction (all files) 2023-04-09 09:55:16,502 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bf27154b49f0831beaa90741ce575bc9/B in TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:16,502 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/0f73b8ad946d4199b91fddf0bc874460, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/c0aa63628b4548bba9707d3173ed0b3f, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/d6a39d3d76dd49cbac4b5fe71c4bc827] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp, totalSize=32.5 K 2023-04-09 09:55:16,502 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 7d245810b3aa4e10a7b61f5caa0e04c8, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1681034114306 2023-04-09 09:55:16,503 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 0f73b8ad946d4199b91fddf0bc874460, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1681034114255 2023-04-09 09:55:16,503 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 27f622741ce54d2c85f56052891edd91, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1681034115415 2023-04-09 09:55:16,503 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting c0aa63628b4548bba9707d3173ed0b3f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1681034114306 2023-04-09 09:55:16,503 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting d6a39d3d76dd49cbac4b5fe71c4bc827, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1681034115415 2023-04-09 09:55:16,508 DEBUG [Thread-2431] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:16,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:16,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:16,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:16,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:16,511 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:16,511 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:16,511 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:16,511 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:16,512 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:16,512 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:16,513 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,513 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:16,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:16,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:16,514 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,514 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:16,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:16,517 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): bf27154b49f0831beaa90741ce575bc9#B#compaction#337 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:55:16,521 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] throttle.PressureAwareThroughputController(145): bf27154b49f0831beaa90741ce575bc9#A#compaction#336 average throughput is 2.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:55:16,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on bf27154b49f0831beaa90741ce575bc9 2023-04-09 09:55:16,523 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bf27154b49f0831beaa90741ce575bc9 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-09 09:55:16,523 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf27154b49f0831beaa90741ce575bc9, store=A 2023-04-09 09:55:16,523 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:55:16,523 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf27154b49f0831beaa90741ce575bc9, store=B 2023-04-09 09:55:16,523 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:16,523 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:55:16,523 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf27154b49f0831beaa90741ce575bc9, store=C 2023-04-09 09:55:16,523 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:55:16,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:16,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:16,524 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:16,529 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:16,529 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:16,529 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:16,529 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:16,529 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:16,532 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:16,532 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:16,532 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:16,532 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. started... 2023-04-09 09:55:16,532 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. as already flushing 2023-04-09 09:55:16,532 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:16,532 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:16,532 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:16,532 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,535 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:16,535 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:16,537 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,537 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:16,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:16,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:16,537 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:16,537 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:16,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:16,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:16,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:16,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:16,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:16,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e0dc183[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:16,540 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:16,540 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:16,542 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,542 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:16,542 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:16,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:16,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:16,543 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:16,543 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:16,543 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:16,544 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:16,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32856 deadline: 1681034176536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,545 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:16,545 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:16,545 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,545 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:16,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:16,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:16,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:16,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:16,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:16,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:16,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:16,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:16,550 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:16,550 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:16,550 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:16,550 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:16,550 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:16,550 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:16,551 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304093eeae94c400b475795dc90d950346d31 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304093eeae94c400b475795dc90d950346d31 2023-04-09 09:55:16,551 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:16,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:16,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:16,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:16,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:16,552 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:16,552 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,552 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:16,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32788 deadline: 1681034176543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:16,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:16,552 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:16,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:16,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:16,552 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:16,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32882 deadline: 1681034176543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:16,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:16,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:16,553 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:16,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:16,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:16,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,557 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,557 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:16,557 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:16,557 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,557 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:16,557 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:16,557 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:16,557 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:16,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:16,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:16,557 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:16,557 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:16,557 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:16,557 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:16,557 DEBUG [Thread-2431] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:16,557 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:16,557 DEBUG [Thread-2431] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:16,557 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=100, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/A/9a866099139546a1986cc0f5e4877c3c 2023-04-09 09:55:16,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:16,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:16,650 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:16,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32856 deadline: 1681034176645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,658 DEBUG [Thread-2431] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:16,658 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:16,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32788 deadline: 1681034176653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,658 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:16,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32882 deadline: 1681034176654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:16,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:16,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:16,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:16,661 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:16,661 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:16,661 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:16,661 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:16,661 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:16,662 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:16,663 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,663 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:16,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:16,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:16,663 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,663 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:16,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:16,663 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:16,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:16,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:16,664 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:16,664 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:16,664 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:16,664 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:16,664 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:16,664 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:16,664 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:16,664 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:16,664 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. started... 2023-04-09 09:55:16,665 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. as already flushing 2023-04-09 09:55:16,665 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:16,665 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:16,665 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:16,665 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,665 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:16,665 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:16,666 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:16,666 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:16,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:16,666 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:16,666 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:16,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:16,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:16,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:16,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:16,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:16,668 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:16,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@395b27b3[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:16,668 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:16,669 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:16,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:16,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:16,669 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:16,669 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:16,669 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:16,669 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,670 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:16,671 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:16,672 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:16,671 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,672 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:16,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:16,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:16,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:16,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:16,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:16,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:16,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:16,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:16,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:16,674 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:16,674 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:16,674 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:16,674 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:16,674 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:16,676 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:16,676 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:16,676 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:16,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:16,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:16,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:16,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:16,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:16,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:16,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:16,678 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:16,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:16,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:16,680 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:16,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:16,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:16,685 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,685 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:16,685 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:16,685 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:16,685 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:16,685 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:16,685 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:16,686 DEBUG [Thread-2431] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:16,686 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:16,685 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:16,686 DEBUG [Thread-2431] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:16,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:16,685 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:16,685 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:16,686 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:16,686 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:16,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:16,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:16,786 DEBUG [Thread-2431] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:16,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:16,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:16,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:16,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:16,789 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:16,790 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:16,790 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:16,790 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:16,790 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:16,790 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:16,791 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,791 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:16,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:16,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:16,791 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,791 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:16,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:16,792 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:16,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:16,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:16,792 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:16,793 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:16,793 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:16,793 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:16,793 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:16,793 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:16,793 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:16,793 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:16,793 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:16,793 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. started... 2023-04-09 09:55:16,793 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. as already flushing 2023-04-09 09:55:16,793 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:16,793 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:16,793 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:16,793 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,793 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:16,793 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:16,795 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,795 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:16,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:16,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:16,795 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:16,795 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:16,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:16,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:16,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:16,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:16,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:16,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5cf16060[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:16,797 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:16,797 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:16,799 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:16,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:16,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:16,799 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:16,799 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:16,799 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,799 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:16,799 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:16,800 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,800 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:16,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:16,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:16,800 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:16,801 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:16,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:16,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:16,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:16,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:16,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:16,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:16,803 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:16,803 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:16,803 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:16,803 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:16,803 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:16,803 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:16,804 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:16,804 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:16,804 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:16,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:16,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:16,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:16,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:16,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:16,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:16,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:16,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:16,805 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:16,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:16,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:16,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:16,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,806 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:16,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:16,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,808 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:16,808 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:16,808 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:16,808 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:16,808 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:16,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:16,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:16,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:16,808 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:16,809 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,809 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:16,809 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:16,809 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:16,809 DEBUG [Thread-2431] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:16,809 DEBUG [Thread-2431] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:16,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:16,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:16,856 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:16,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32856 deadline: 1681034176851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,863 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:16,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32788 deadline: 1681034176859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,864 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:16,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32882 deadline: 1681034176860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,909 DEBUG [Thread-2431] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:16,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:16,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:16,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:16,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:16,912 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:16,913 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:16,913 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:16,913 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:16,913 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:16,913 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:16,917 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,917 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:16,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:16,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:16,917 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,917 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:16,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:16,918 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:16,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:16,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:16,918 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:16,918 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:16,918 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:16,918 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:16,919 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:16,919 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:16,920 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:16,920 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:16,920 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:16,920 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. started... 2023-04-09 09:55:16,920 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. as already flushing 2023-04-09 09:55:16,921 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:16,921 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:16,921 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:16,921 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,921 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:16,921 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:16,922 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:16,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:16,922 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:16,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:16,922 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:16,922 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:16,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:16,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:16,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:16,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:16,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38333823[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:16,923 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:16,924 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:16,924 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:16,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:16,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:16,925 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:16,925 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:16,925 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,925 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:16,925 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:16,926 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,926 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:16,926 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:16,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,926 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:16,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:16,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:16,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:16,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:16,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:16,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:16,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:16,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:16,928 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:16,928 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:16,928 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:16,928 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:16,928 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:16,928 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:16,930 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:16,930 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:16,930 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:16,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:16,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:16,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:16,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:16,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:16,930 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:16,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:16,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:16,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:16,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:16,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:16,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:16,932 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:16,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:16,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,934 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:16,934 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:16,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:16,934 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:16,934 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:16,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:16,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:16,935 DEBUG [Thread-2431] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:16,934 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:16,935 DEBUG [Thread-2431] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:16,935 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:16,935 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:16,935 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:16,935 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:16,935 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:16,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:16,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:16,935 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:16,940 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/B/ef311de467a94c84bb1bc30e5b4a83be as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/ef311de467a94c84bb1bc30e5b4a83be 2023-04-09 09:55:16,944 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bf27154b49f0831beaa90741ce575bc9/B of bf27154b49f0831beaa90741ce575bc9 into ef311de467a94c84bb1bc30e5b4a83be(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:55:16,944 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bf27154b49f0831beaa90741ce575bc9: 2023-04-09 09:55:16,944 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9., storeName=bf27154b49f0831beaa90741ce575bc9/B, priority=13, startTime=1681034116500; duration=0sec 2023-04-09 09:55:16,944 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:55:16,944 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-09 09:55:16,945 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-09 09:55:16,945 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): bf27154b49f0831beaa90741ce575bc9/C is initiating minor compaction (all files) 2023-04-09 09:55:16,945 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bf27154b49f0831beaa90741ce575bc9/C in TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:16,945 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/65ccd0efd6894cbc93de631d4f2b993f, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/c3d6fa61659e4dd5a374c671526fbb25, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/5bd2e8b8bdc34f83a64e17fa582ef1d8] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp, totalSize=32.5 K 2023-04-09 09:55:16,945 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 65ccd0efd6894cbc93de631d4f2b993f, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1681034114255 2023-04-09 09:55:16,946 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting c3d6fa61659e4dd5a374c671526fbb25, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1681034114306 2023-04-09 09:55:16,946 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 5bd2e8b8bdc34f83a64e17fa582ef1d8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1681034115415 2023-04-09 09:55:16,954 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-09 09:55:16,963 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): bf27154b49f0831beaa90741ce575bc9#C#compaction#340 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:55:16,968 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=100 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/B/9ac2a805efa14c969071a482eb009c65 2023-04-09 09:55:17,001 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/C/22d3b8e59d0443bf9e44a9bfbe2d9778 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/22d3b8e59d0443bf9e44a9bfbe2d9778 2023-04-09 09:55:17,007 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bf27154b49f0831beaa90741ce575bc9/C of bf27154b49f0831beaa90741ce575bc9 into 22d3b8e59d0443bf9e44a9bfbe2d9778(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:55:17,007 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bf27154b49f0831beaa90741ce575bc9: 2023-04-09 09:55:17,007 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9., storeName=bf27154b49f0831beaa90741ce575bc9/C, priority=13, startTime=1681034116500; duration=0sec 2023-04-09 09:55:17,007 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:55:17,015 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=100 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/C/655b866eaea04f59b092b653ce7cf38c 2023-04-09 09:55:17,019 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/A/9a866099139546a1986cc0f5e4877c3c as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/9a866099139546a1986cc0f5e4877c3c 2023-04-09 09:55:17,024 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/9a866099139546a1986cc0f5e4877c3c, entries=200, sequenceid=100, filesize=31.9 K 2023-04-09 09:55:17,025 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/B/9ac2a805efa14c969071a482eb009c65 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/9ac2a805efa14c969071a482eb009c65 2023-04-09 09:55:17,028 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/9ac2a805efa14c969071a482eb009c65, entries=150, sequenceid=100, filesize=11.6 K 2023-04-09 09:55:17,029 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/C/655b866eaea04f59b092b653ce7cf38c as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/655b866eaea04f59b092b653ce7cf38c 2023-04-09 09:55:17,032 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/655b866eaea04f59b092b653ce7cf38c, entries=150, sequenceid=100, filesize=11.6 K 2023-04-09 09:55:17,033 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for bf27154b49f0831beaa90741ce575bc9 in 509ms, sequenceid=100, compaction requested=false 2023-04-09 09:55:17,033 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bf27154b49f0831beaa90741ce575bc9: 2023-04-09 09:55:17,035 DEBUG [Thread-2431] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:17,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:17,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:17,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:17,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:17,048 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:17,048 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:17,048 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:17,048 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:17,049 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:17,049 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:17,050 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:17,050 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:17,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:17,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:17,050 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:17,050 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:17,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:17,051 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:17,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:17,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:17,051 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:17,051 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:17,051 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:17,051 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:17,051 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:17,051 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:17,053 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:17,054 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:17,054 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:17,054 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. started... 2023-04-09 09:55:17,054 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bf27154b49f0831beaa90741ce575bc9 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-09 09:55:17,054 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf27154b49f0831beaa90741ce575bc9, store=A 2023-04-09 09:55:17,054 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:17,054 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf27154b49f0831beaa90741ce575bc9, store=B 2023-04-09 09:55:17,054 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:17,054 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf27154b49f0831beaa90741ce575bc9, store=C 2023-04-09 09:55:17,054 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:17,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on bf27154b49f0831beaa90741ce575bc9 2023-04-09 09:55:17,162 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. as already flushing 2023-04-09 09:55:17,218 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681034117188, max timestamp=1681034117216Num uniques 859; 2023-04-09 09:55:17,218 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681034117188, max timestamp=1681034117216Num uniques 859; 2023-04-09 09:55:17,219 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-09 09:55:17,218 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681034117188, max timestamp=1681034117216Num uniques 859; 2023-04-09 09:55:17,218 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-09 09:55:17,219 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-09 09:55:17,286 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:17,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32788 deadline: 1681034177280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:17,286 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:17,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32856 deadline: 1681034177280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:17,286 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:17,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32882 deadline: 1681034177280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:17,387 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/A/07f0337b87d447b0b75cb75da29dacee as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/07f0337b87d447b0b75cb75da29dacee 2023-04-09 09:55:17,391 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bf27154b49f0831beaa90741ce575bc9/A of bf27154b49f0831beaa90741ce575bc9 into 07f0337b87d447b0b75cb75da29dacee(size=25.2 K), total size for store is 57.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:55:17,391 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bf27154b49f0831beaa90741ce575bc9: 2023-04-09 09:55:17,391 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9., storeName=bf27154b49f0831beaa90741ce575bc9/A, priority=13, startTime=1681034116500; duration=0sec 2023-04-09 09:55:17,391 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:55:17,393 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:17,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32788 deadline: 1681034177387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:17,393 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:17,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32856 deadline: 1681034177387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:17,394 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:17,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32882 deadline: 1681034177387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:17,471 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040958cd78fcf9e649928f69a4cd2e46a4c4 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040958cd78fcf9e649928f69a4cd2e46a4c4 2023-04-09 09:55:17,475 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=112, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/A/f19f8e87c9d441a6814e7e412dbbda24 2023-04-09 09:55:17,598 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:17,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32856 deadline: 1681034177594, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:17,601 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:17,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32788 deadline: 1681034177595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:17,601 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:17,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32882 deadline: 1681034177595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:17,884 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=112 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/B/1e70a4e5eec94a2cbd07796c7d20a556 2023-04-09 09:55:17,906 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:17,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32856 deadline: 1681034177900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:17,907 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:17,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32788 deadline: 1681034177903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:17,909 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:17,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32882 deadline: 1681034177904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,295 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=112 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/C/a14e2ebb4b2643ca84ade277d9bd11e3 2023-04-09 09:55:18,300 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/A/f19f8e87c9d441a6814e7e412dbbda24 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/f19f8e87c9d441a6814e7e412dbbda24 2023-04-09 09:55:18,304 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/f19f8e87c9d441a6814e7e412dbbda24, entries=150, sequenceid=112, filesize=25.1 K 2023-04-09 09:55:18,305 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/B/1e70a4e5eec94a2cbd07796c7d20a556 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/1e70a4e5eec94a2cbd07796c7d20a556 2023-04-09 09:55:18,311 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/1e70a4e5eec94a2cbd07796c7d20a556, entries=150, sequenceid=112, filesize=11.6 K 2023-04-09 09:55:18,312 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/C/a14e2ebb4b2643ca84ade277d9bd11e3 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/a14e2ebb4b2643ca84ade277d9bd11e3 2023-04-09 09:55:18,315 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/a14e2ebb4b2643ca84ade277d9bd11e3, entries=150, sequenceid=112, filesize=11.6 K 2023-04-09 09:55:18,316 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=147.60 KB/151140 for bf27154b49f0831beaa90741ce575bc9 in 1262ms, sequenceid=112, compaction requested=true 2023-04-09 09:55:18,316 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bf27154b49f0831beaa90741ce575bc9: 2023-04-09 09:55:18,316 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:18,316 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:18,316 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:18,316 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,317 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:18,317 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:18,318 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:18,318 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:18,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:18,318 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:18,318 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:18,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:18,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:18,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:18,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:18,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:18,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@882e77f[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:18,321 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:18,321 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:18,323 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:18,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:18,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:18,323 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:18,323 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,323 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:18,323 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:18,323 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:18,325 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,325 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:18,325 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:18,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:18,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:18,325 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:18,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:18,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:18,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:18,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:18,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:18,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:18,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:18,328 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:18,328 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:18,328 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:18,328 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:18,328 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:18,329 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:18,329 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:18,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:18,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:18,329 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:18,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:18,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:18,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:18,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:18,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:18,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:18,330 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:18,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:18,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:18,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:18,336 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:18,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:18,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,342 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,342 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:18,342 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:18,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:18,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:18,342 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:18,342 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:18,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:18,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:18,342 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:18,342 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:18,343 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:18,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:18,343 DEBUG [Thread-2431] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:18,343 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:18,343 DEBUG [Thread-2431] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:18,343 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:18,343 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:18,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on bf27154b49f0831beaa90741ce575bc9 2023-04-09 09:55:18,416 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bf27154b49f0831beaa90741ce575bc9 3/3 column families, dataSize=154.31 KB heapSize=430.69 KB 2023-04-09 09:55:18,417 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf27154b49f0831beaa90741ce575bc9, store=A 2023-04-09 09:55:18,417 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-09 09:55:18,417 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf27154b49f0831beaa90741ce575bc9, store=B 2023-04-09 09:55:18,417 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-09 09:55:18,417 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf27154b49f0831beaa90741ce575bc9, store=C 2023-04-09 09:55:18,417 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-09 09:55:18,427 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:18,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32856 deadline: 1681034178422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,430 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230409f9924598a25446baa567bdd665634c44 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409f9924598a25446baa567bdd665634c44 2023-04-09 09:55:18,433 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:18,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32788 deadline: 1681034178426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,433 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:18,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32882 deadline: 1681034178427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,443 DEBUG [Thread-2431] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:18,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:18,444 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=151, memsize=53.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/A/7387c8f2f2694f2b8583c1145a72f45c 2023-04-09 09:55:18,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:18,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:18,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:18,446 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:18,446 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:18,446 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:18,446 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:18,447 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:18,447 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:18,449 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:18,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32772 deadline: 1681034178443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,449 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:18,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:18,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:18,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:18,451 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:18,451 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,452 DEBUG [Thread-2425] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4143 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9., hostname=jenkins-hbase4.apache.org,42467,1681034019544, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-09 09:55:18,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:18,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:18,453 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:18,453 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,453 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:18,456 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:18,456 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:18,456 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:18,456 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:18,456 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:18,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32870 deadline: 1681034178447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,456 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:18,472 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:18,474 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:18,474 DEBUG [Thread-2427] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4148 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9., hostname=jenkins-hbase4.apache.org,42467,1681034019544, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-09 09:55:18,480 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:18,480 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. started... 2023-04-09 09:55:18,480 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. as already flushing 2023-04-09 09:55:18,480 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:18,480 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:18,480 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:18,480 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,484 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:18,484 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:18,485 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:18,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:18,485 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:18,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:18,486 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:18,486 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:18,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:18,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:18,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:18,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:18,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68842f63[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:18,488 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:18,488 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:18,490 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:18,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:18,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:18,490 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,490 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:18,490 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:18,490 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:18,490 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:18,492 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:18,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:18,492 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:18,492 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:18,492 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:18,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:18,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:18,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:18,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:18,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:18,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:18,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:18,495 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:18,496 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:18,496 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:18,496 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:18,496 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:18,498 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:18,498 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:18,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:18,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:18,498 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:18,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:18,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:18,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:18,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:18,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:18,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:18,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:18,499 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:18,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:18,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:18,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,501 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:18,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:18,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,503 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,504 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:18,504 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:18,504 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:18,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:18,504 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,504 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:18,504 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:18,504 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:18,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:18,504 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:18,504 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:18,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:18,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:18,504 DEBUG [Thread-2431] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:18,504 DEBUG [Thread-2431] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:18,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:18,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:18,537 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:18,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32856 deadline: 1681034178528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,542 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:18,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32882 deadline: 1681034178534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,542 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:18,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32788 deadline: 1681034178535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,604 DEBUG [Thread-2431] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:18,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:18,608 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:18,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:18,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:18,609 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:18,609 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:18,610 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:18,610 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:18,611 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:18,611 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:18,614 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,614 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:18,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:18,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:18,614 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,614 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:18,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:18,615 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:18,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:18,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:18,615 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:18,621 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:18,621 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:18,621 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:18,621 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:18,622 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:18,624 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:18,624 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:18,624 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:18,624 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. started... 2023-04-09 09:55:18,624 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. as already flushing 2023-04-09 09:55:18,624 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:18,624 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:18,624 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:18,624 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,624 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:18,624 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:18,626 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,626 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:18,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:18,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:18,627 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:18,627 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:18,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:18,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:18,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:18,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:18,628 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:18,628 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3fa1a073[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:18,629 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:18,629 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:18,630 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:18,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:18,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:18,631 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:18,631 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:18,631 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:18,631 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,631 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:18,632 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,632 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:18,632 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:18,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:18,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:18,632 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:18,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:18,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:18,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:18,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:18,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:18,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:18,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:18,638 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:18,638 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:18,638 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:18,638 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:18,638 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:18,646 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:18,646 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:18,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:18,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:18,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:18,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:18,646 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:18,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:18,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:18,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:18,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:18,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:18,648 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:18,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:18,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:18,649 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:18,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:18,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,653 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:18,653 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:18,653 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,653 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:18,653 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:18,653 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:18,653 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,653 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:18,653 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:18,654 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:18,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:18,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:18,654 DEBUG [Thread-2431] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:18,654 DEBUG [Thread-2431] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:18,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:18,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:18,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:18,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:18,740 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:18,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32856 deadline: 1681034178739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,751 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:18,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32882 deadline: 1681034178743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,751 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:18,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32788 deadline: 1681034178743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,754 DEBUG [Thread-2431] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:18,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:18,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:18,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:18,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:18,757 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:18,757 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:18,757 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:18,757 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:18,758 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:18,758 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:18,759 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,759 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:18,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:18,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:18,759 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:18,759 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:18,760 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:18,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:18,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:18,760 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:18,760 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:18,760 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:18,760 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:18,760 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:18,761 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:18,761 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:18,761 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:18,761 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:18,761 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. started... 2023-04-09 09:55:18,761 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. as already flushing 2023-04-09 09:55:18,761 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:18,761 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:18,761 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:18,761 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,761 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:18,761 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:18,762 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:18,762 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:18,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:18,763 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:18,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:18,763 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:18,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:18,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:18,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:18,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:18,764 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:18,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ef27584[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:18,764 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:18,766 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:18,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:18,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:18,766 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,766 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:18,767 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:18,766 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:18,767 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:18,768 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,768 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:18,768 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:18,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:18,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:18,768 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:18,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:18,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:18,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:18,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:18,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:18,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:18,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:18,770 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:18,770 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:18,770 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:18,770 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:18,770 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:18,771 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:18,771 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:18,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:18,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:18,771 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:18,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:18,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:18,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:18,772 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:18,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:18,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:18,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:18,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:18,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:18,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:18,773 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:18,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:18,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,775 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,775 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:18,775 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:18,775 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:18,775 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:18,775 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:18,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:18,775 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:18,775 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:18,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:18,776 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:18,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:18,776 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:18,776 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:18,776 DEBUG [Thread-2431] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:18,776 DEBUG [Thread-2431] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:18,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:18,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:18,876 DEBUG [Thread-2431] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:18,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:18,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:18,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:18,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:18,880 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:18,880 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:18,880 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:18,880 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:18,880 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:18,880 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:18,885 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:18,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:18,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:18,887 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:18,887 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,887 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:18,887 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:18,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:18,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:18,888 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:18,888 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:18,888 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:18,888 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:18,888 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:18,888 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:18,888 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:18,889 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:18,889 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:18,889 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. started... 2023-04-09 09:55:18,889 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. as already flushing 2023-04-09 09:55:18,889 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:18,890 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:18,890 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:18,890 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,890 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:18,890 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:18,890 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=151 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/B/f0829384229a4f94841c271a93b58ca3 2023-04-09 09:55:18,892 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:18,892 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:18,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:18,893 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:18,893 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:18,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:18,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:18,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:18,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:18,895 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:18,895 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d4c03a[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:18,895 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:18,895 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:18,896 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:18,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:18,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:18,896 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,896 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:18,898 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:18,898 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:18,898 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:18,900 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:18,900 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:18,900 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:18,900 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:18,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:18,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:18,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:18,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:18,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:18,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:18,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:18,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:18,911 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:18,912 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:18,912 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:18,912 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:18,912 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:18,915 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:18,915 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:18,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:18,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:18,915 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:18,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:18,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:18,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:18,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:18,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:18,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:18,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:18,916 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:18,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:18,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:18,917 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:18,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:18,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,919 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:18,919 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:18,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:18,919 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:18,919 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:18,919 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:18,919 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:18,920 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:18,919 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:18,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:18,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:18,920 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:18,920 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:18,920 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:18,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:18,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:18,920 DEBUG [Thread-2431] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:18,920 DEBUG [Thread-2431] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:19,020 DEBUG [Thread-2431] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:19,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:19,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:19,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:19,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:19,024 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:19,024 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:19,024 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:19,024 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:19,024 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:19,024 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:19,025 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,025 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:19,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:19,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:19,026 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,026 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:19,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:19,026 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:19,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:19,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:19,027 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:19,028 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:19,028 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:19,028 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:19,028 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:19,028 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:19,028 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:19,028 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:19,028 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:19,028 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. started... 2023-04-09 09:55:19,028 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. as already flushing 2023-04-09 09:55:19,028 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:19,028 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:19,029 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:19,029 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,029 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:19,029 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:19,030 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:19,030 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:19,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:19,030 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:19,030 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:19,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:19,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:19,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:19,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:19,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:19,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48e20854[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:19,032 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:19,032 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:19,034 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:19,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:19,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:19,034 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:19,035 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:19,035 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:19,035 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,035 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:19,036 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,036 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:19,036 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:19,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:19,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:19,036 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:19,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:19,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:19,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:19,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:19,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:19,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:19,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:19,039 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:19,039 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:19,039 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:19,039 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:19,039 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:19,040 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:19,040 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:19,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:19,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:19,040 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:19,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:19,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:19,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:19,041 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:19,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:19,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:19,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:19,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:19,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:19,043 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:19,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:19,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:19,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,045 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:19,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32856 deadline: 1681034179041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,045 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,045 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:19,045 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:19,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:19,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:19,045 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:19,046 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:19,046 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:19,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:19,046 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:19,046 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:19,046 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:19,046 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:19,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:19,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:19,046 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:19,046 DEBUG [Thread-2431] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:19,046 DEBUG [Thread-2431] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:19,057 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:19,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32882 deadline: 1681034179052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,057 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:19,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32788 deadline: 1681034179052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,147 DEBUG [Thread-2431] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:19,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:19,149 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:19,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:19,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:19,150 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:19,150 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:19,150 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:19,150 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:19,150 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:19,151 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:19,157 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:19,157 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:19,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:19,157 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,157 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:19,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:19,158 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:19,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:19,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:19,158 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:19,158 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:19,158 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:19,159 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:19,159 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:19,159 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:19,159 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:19,159 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:19,159 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:19,159 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. started... 2023-04-09 09:55:19,159 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. as already flushing 2023-04-09 09:55:19,159 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:19,159 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:19,159 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:19,159 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,159 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:19,159 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:19,161 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,161 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:19,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:19,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:19,161 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:19,161 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:19,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:19,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:19,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:19,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:19,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:19,162 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:19,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a728b03[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:19,162 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:19,163 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:19,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:19,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:19,163 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:19,163 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:19,164 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:19,165 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,165 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:19,166 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,166 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:19,166 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:19,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:19,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:19,166 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:19,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:19,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:19,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:19,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:19,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:19,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:19,168 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:19,168 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:19,168 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:19,168 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:19,168 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:19,168 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:19,171 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:19,171 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:19,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:19,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:19,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:19,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:19,171 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:19,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:19,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:19,172 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:19,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:19,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:19,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:19,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:19,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:19,173 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:19,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:19,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,178 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,178 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:19,179 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:19,179 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,179 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:19,179 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:19,179 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:19,178 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:19,179 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:19,179 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:19,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:19,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:19,179 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:19,179 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:19,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:19,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:19,179 DEBUG [Thread-2431] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:19,179 DEBUG [Thread-2431] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:19,279 DEBUG [Thread-2431] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:19,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:19,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:19,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:19,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:19,283 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:19,283 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:19,283 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:19,283 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:19,283 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:19,283 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:19,284 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:19,284 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:19,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:19,284 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,285 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:19,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:19,285 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:19,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:19,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:19,285 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:19,285 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:19,285 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:19,286 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:19,286 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:19,286 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:19,286 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:19,286 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:19,286 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:19,286 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. started... 2023-04-09 09:55:19,286 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. as already flushing 2023-04-09 09:55:19,286 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:19,286 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:19,286 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:19,286 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,286 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:19,286 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:19,291 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:19,291 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:19,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:19,291 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:19,291 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:19,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:19,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:19,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:19,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:19,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:19,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77219f9c[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:19,293 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:19,293 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:19,294 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:19,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:19,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:19,294 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:19,294 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:19,294 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:19,295 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,295 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:19,295 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:19,295 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,295 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:19,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:19,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:19,296 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:19,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:19,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:19,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:19,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:19,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:19,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:19,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:19,298 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:19,298 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:19,298 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:19,298 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:19,298 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:19,299 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:19,299 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:19,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:19,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:19,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:19,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:19,299 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:19,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:19,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:19,300 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:19,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:19,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:19,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:19,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:19,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:19,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,301 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:19,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:19,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,304 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:19,304 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:19,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:19,304 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:19,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:19,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:19,304 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:19,304 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:19,305 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:19,305 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,305 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:19,305 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:19,305 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:19,305 DEBUG [Thread-2431] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:19,305 DEBUG [Thread-2431] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:19,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:19,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:19,326 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=151 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/C/29e6b10df77e46d4b6832898ee7eab53 2023-04-09 09:55:19,330 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/A/7387c8f2f2694f2b8583c1145a72f45c as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/7387c8f2f2694f2b8583c1145a72f45c 2023-04-09 09:55:19,333 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/7387c8f2f2694f2b8583c1145a72f45c, entries=250, sequenceid=151, filesize=38.9 K 2023-04-09 09:55:19,333 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/B/f0829384229a4f94841c271a93b58ca3 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/f0829384229a4f94841c271a93b58ca3 2023-04-09 09:55:19,337 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/f0829384229a4f94841c271a93b58ca3, entries=150, sequenceid=151, filesize=11.8 K 2023-04-09 09:55:19,338 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/C/29e6b10df77e46d4b6832898ee7eab53 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/29e6b10df77e46d4b6832898ee7eab53 2023-04-09 09:55:19,341 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/29e6b10df77e46d4b6832898ee7eab53, entries=150, sequenceid=151, filesize=11.8 K 2023-04-09 09:55:19,342 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~161.02 KB/164880, heapSize ~451.27 KB/462096, currentSize=26.84 KB/27480 for bf27154b49f0831beaa90741ce575bc9 in 926ms, sequenceid=151, compaction requested=true 2023-04-09 09:55:19,342 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bf27154b49f0831beaa90741ce575bc9: 2023-04-09 09:55:19,342 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:55:19,342 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-09 09:55:19,342 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-09 09:55:19,342 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:55:19,342 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:55:19,343 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 124161 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-09 09:55:19,344 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47797 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-09 09:55:19,344 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1912): bf27154b49f0831beaa90741ce575bc9/A is initiating minor compaction (all files) 2023-04-09 09:55:19,344 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): bf27154b49f0831beaa90741ce575bc9/B is initiating minor compaction (all files) 2023-04-09 09:55:19,344 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bf27154b49f0831beaa90741ce575bc9/A in TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:19,344 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bf27154b49f0831beaa90741ce575bc9/B in TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:19,344 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/07f0337b87d447b0b75cb75da29dacee, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/9a866099139546a1986cc0f5e4877c3c, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/f19f8e87c9d441a6814e7e412dbbda24, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/7387c8f2f2694f2b8583c1145a72f45c] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp, totalSize=121.3 K 2023-04-09 09:55:19,344 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/ef311de467a94c84bb1bc30e5b4a83be, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/9ac2a805efa14c969071a482eb009c65, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/1e70a4e5eec94a2cbd07796c7d20a556, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/f0829384229a4f94841c271a93b58ca3] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp, totalSize=46.7 K 2023-04-09 09:55:19,345 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 07f0337b87d447b0b75cb75da29dacee, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1681034115415 2023-04-09 09:55:19,345 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting ef311de467a94c84bb1bc30e5b4a83be, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1681034115415 2023-04-09 09:55:19,345 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 9a866099139546a1986cc0f5e4877c3c, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=100, earliestPutTs=1681034116202 2023-04-09 09:55:19,345 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 9ac2a805efa14c969071a482eb009c65, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=100, earliestPutTs=1681034116207 2023-04-09 09:55:19,345 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting f19f8e87c9d441a6814e7e412dbbda24, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=112, earliestPutTs=1681034116534 2023-04-09 09:55:19,346 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 1e70a4e5eec94a2cbd07796c7d20a556, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=112, earliestPutTs=1681034116534 2023-04-09 09:55:19,346 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 7387c8f2f2694f2b8583c1145a72f45c, keycount=250, bloomtype=ROW, size=38.9 K, encoding=NONE, compression=NONE, seqNum=151, earliestPutTs=1681034117233 2023-04-09 09:55:19,346 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting f0829384229a4f94841c271a93b58ca3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=151, earliestPutTs=1681034117260 2023-04-09 09:55:19,356 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): bf27154b49f0831beaa90741ce575bc9#B#compaction#349 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:55:19,378 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] throttle.PressureAwareThroughputController(145): bf27154b49f0831beaa90741ce575bc9#A#compaction#348 average throughput is 0.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:55:19,405 DEBUG [Thread-2431] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:19,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:19,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:19,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:19,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:19,430 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:19,430 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:19,430 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:19,430 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:19,430 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:19,430 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:19,431 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:19,431 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:19,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:19,432 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:19,432 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:19,432 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:19,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:19,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:19,433 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:19,444 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:19,445 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:19,445 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:19,445 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:19,445 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:19,446 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:19,446 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:19,446 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:19,446 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. started... 2023-04-09 09:55:19,446 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bf27154b49f0831beaa90741ce575bc9 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-09 09:55:19,446 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf27154b49f0831beaa90741ce575bc9, store=A 2023-04-09 09:55:19,446 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:19,446 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf27154b49f0831beaa90741ce575bc9, store=B 2023-04-09 09:55:19,446 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:19,447 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf27154b49f0831beaa90741ce575bc9, store=C 2023-04-09 09:55:19,447 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:19,450 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/B/173860f44f104ca3a42cfc17f917511f as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/173860f44f104ca3a42cfc17f917511f 2023-04-09 09:55:19,468 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in bf27154b49f0831beaa90741ce575bc9/B of bf27154b49f0831beaa90741ce575bc9 into 173860f44f104ca3a42cfc17f917511f(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:55:19,468 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bf27154b49f0831beaa90741ce575bc9: 2023-04-09 09:55:19,468 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9., storeName=bf27154b49f0831beaa90741ce575bc9/B, priority=12, startTime=1681034119342; duration=0sec 2023-04-09 09:55:19,468 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:55:19,468 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-09 09:55:19,470 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47797 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-09 09:55:19,470 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): bf27154b49f0831beaa90741ce575bc9/C is initiating minor compaction (all files) 2023-04-09 09:55:19,470 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bf27154b49f0831beaa90741ce575bc9/C in TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:19,470 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/22d3b8e59d0443bf9e44a9bfbe2d9778, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/655b866eaea04f59b092b653ce7cf38c, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/a14e2ebb4b2643ca84ade277d9bd11e3, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/29e6b10df77e46d4b6832898ee7eab53] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp, totalSize=46.7 K 2023-04-09 09:55:19,471 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 22d3b8e59d0443bf9e44a9bfbe2d9778, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1681034115415 2023-04-09 09:55:19,471 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 655b866eaea04f59b092b653ce7cf38c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=100, earliestPutTs=1681034116207 2023-04-09 09:55:19,472 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting a14e2ebb4b2643ca84ade277d9bd11e3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=112, earliestPutTs=1681034116534 2023-04-09 09:55:19,472 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 29e6b10df77e46d4b6832898ee7eab53, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=151, earliestPutTs=1681034117260 2023-04-09 09:55:19,476 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/A/e0cbce6fc27c484bbc7a8b518327e484 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/e0cbce6fc27c484bbc7a8b518327e484 2023-04-09 09:55:19,481 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in bf27154b49f0831beaa90741ce575bc9/A of bf27154b49f0831beaa90741ce575bc9 into e0cbce6fc27c484bbc7a8b518327e484(size=25.5 K), total size for store is 25.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:55:19,481 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bf27154b49f0831beaa90741ce575bc9: 2023-04-09 09:55:19,481 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9., storeName=bf27154b49f0831beaa90741ce575bc9/A, priority=12, startTime=1681034119342; duration=0sec 2023-04-09 09:55:19,481 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): bf27154b49f0831beaa90741ce575bc9#C#compaction#351 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:55:19,481 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:55:19,510 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/C/b7d65a78164240969fdafee199635db2 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/b7d65a78164240969fdafee199635db2 2023-04-09 09:55:19,514 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in bf27154b49f0831beaa90741ce575bc9/C of bf27154b49f0831beaa90741ce575bc9 into b7d65a78164240969fdafee199635db2(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:55:19,515 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bf27154b49f0831beaa90741ce575bc9: 2023-04-09 09:55:19,515 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9., storeName=bf27154b49f0831beaa90741ce575bc9/C, priority=12, startTime=1681034119342; duration=0sec 2023-04-09 09:55:19,515 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:55:19,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on bf27154b49f0831beaa90741ce575bc9 2023-04-09 09:55:19,573 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. as already flushing 2023-04-09 09:55:19,614 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681034119601, max timestamp=1681034119612Num uniques 859; 2023-04-09 09:55:19,614 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-09 09:55:19,616 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681034119601, max timestamp=1681034119612Num uniques 859; 2023-04-09 09:55:19,616 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-09 09:55:19,617 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681034119601, max timestamp=1681034119612Num uniques 859; 2023-04-09 09:55:19,617 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-09 09:55:19,669 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:19,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32856 deadline: 1681034179662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,669 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681034119637, max timestamp=1681034119667Num uniques 1068; 2023-04-09 09:55:19,669 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-09 09:55:19,670 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681034119637, max timestamp=1681034119667Num uniques 1068; 2023-04-09 09:55:19,670 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-09 09:55:19,670 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681034119637, max timestamp=1681034119667Num uniques 1068; 2023-04-09 09:55:19,670 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-09 09:55:19,711 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681034119688, max timestamp=1681034119709Num uniques 1027; 2023-04-09 09:55:19,711 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-09 09:55:19,711 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681034119688, max timestamp=1681034119709Num uniques 1027; 2023-04-09 09:55:19,711 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-09 09:55:19,711 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681034119688, max timestamp=1681034119709Num uniques 1027; 2023-04-09 09:55:19,711 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-09 09:55:19,751 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681034119731, max timestamp=1681034119749Num uniques 1086; 2023-04-09 09:55:19,751 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-09 09:55:19,751 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681034119731, max timestamp=1681034119749Num uniques 1086; 2023-04-09 09:55:19,751 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-09 09:55:19,751 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681034119731, max timestamp=1681034119749Num uniques 1086; 2023-04-09 09:55:19,752 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-09 09:55:19,782 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1681034119769, max timestamp=1681034119780Num uniques 1095; 2023-04-09 09:55:19,782 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=13510, suffix heap size=141848,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=295 2023-04-09 09:55:19,782 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1681034119769, max timestamp=1681034119780Num uniques 1095; 2023-04-09 09:55:19,782 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=13510, suffix heap size=141848,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=295 2023-04-09 09:55:19,783 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1681034119769, max timestamp=1681034119780Num uniques 1095; 2023-04-09 09:55:19,783 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=13510, suffix heap size=141848,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=295 2023-04-09 09:55:19,841 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:19,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32882 deadline: 1681034179838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,844 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:19,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32856 deadline: 1681034179838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,847 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:19,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32788 deadline: 1681034179839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,872 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230409ce460e1b47084abebbb3fa68bdff5216 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409ce460e1b47084abebbb3fa68bdff5216 2023-04-09 09:55:19,878 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=158, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/A/a64933e6e15d43b39e74cfc55557e4ad 2023-04-09 09:55:19,890 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=158 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/B/9169639d247a48a981b1b7a471bdca44 2023-04-09 09:55:19,899 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=158 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/C/27458d76c0034be3bc60a79ca3306061 2023-04-09 09:55:19,903 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/A/a64933e6e15d43b39e74cfc55557e4ad as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/a64933e6e15d43b39e74cfc55557e4ad 2023-04-09 09:55:19,908 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/a64933e6e15d43b39e74cfc55557e4ad, entries=100, sequenceid=158, filesize=18.5 K 2023-04-09 09:55:19,909 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/B/9169639d247a48a981b1b7a471bdca44 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/9169639d247a48a981b1b7a471bdca44 2023-04-09 09:55:19,912 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/9169639d247a48a981b1b7a471bdca44, entries=100, sequenceid=158, filesize=9.4 K 2023-04-09 09:55:19,913 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/C/27458d76c0034be3bc60a79ca3306061 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/27458d76c0034be3bc60a79ca3306061 2023-04-09 09:55:19,916 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/27458d76c0034be3bc60a79ca3306061, entries=100, sequenceid=158, filesize=9.4 K 2023-04-09 09:55:19,917 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=154.31 KB/158010 for bf27154b49f0831beaa90741ce575bc9 in 471ms, sequenceid=158, compaction requested=false 2023-04-09 09:55:19,917 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bf27154b49f0831beaa90741ce575bc9: 2023-04-09 09:55:19,917 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:19,917 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:19,917 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:19,917 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,917 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:19,917 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:19,919 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,919 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:19,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:19,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:19,919 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:19,919 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:19,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:19,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:19,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:19,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:19,921 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:19,921 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50534452[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:19,921 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:19,921 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:19,924 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:19,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:19,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:19,924 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:19,925 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,925 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:19,925 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:19,925 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:19,927 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:19,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:19,927 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:19,927 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:19,927 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:19,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:19,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:19,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:19,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:19,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:19,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:19,929 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:19,929 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:19,930 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:19,930 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:19,930 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:19,930 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:19,931 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:19,931 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:19,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:19,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:19,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:19,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:19,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:19,932 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:19,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:19,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:19,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:19,932 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:19,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:19,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:19,934 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:19,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:19,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:19,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,936 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:19,936 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:19,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:19,936 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:19,936 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:19,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:19,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:19,936 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:19,936 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:19,936 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:19,936 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:19,936 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:19,936 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:19,936 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:19,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:19,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:19,936 DEBUG [Thread-2431] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:19,936 DEBUG [Thread-2431] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:19,946 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681034119802, max timestamp=1681034119944Num uniques 1154; 2023-04-09 09:55:19,946 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=7050, suffix heap size=147984,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=154 2023-04-09 09:55:19,947 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681034119802, max timestamp=1681034119944Num uniques 1154; 2023-04-09 09:55:19,947 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681034119802, max timestamp=1681034119944Num uniques 1154; 2023-04-09 09:55:19,947 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=7050, suffix heap size=147984,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=154 2023-04-09 09:55:19,947 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=7050, suffix heap size=147984,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=154 2023-04-09 09:55:19,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on bf27154b49f0831beaa90741ce575bc9 2023-04-09 09:55:19,953 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bf27154b49f0831beaa90741ce575bc9 3/3 column families, dataSize=53.67 KB heapSize=146.39 KB 2023-04-09 09:55:19,953 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf27154b49f0831beaa90741ce575bc9, store=A 2023-04-09 09:55:19,953 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:55:19,953 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf27154b49f0831beaa90741ce575bc9, store=B 2023-04-09 09:55:19,953 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:55:19,953 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf27154b49f0831beaa90741ce575bc9, store=C 2023-04-09 09:55:19,953 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:55:19,973 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230409972811e338a74d87a7911c5bef02a268 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409972811e338a74d87a7911c5bef02a268 2023-04-09 09:55:19,985 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=272, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/A/cd46d9c1278c4db296846d0bdadbbf98 2023-04-09 09:55:20,017 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681034119997, max timestamp=1681034120015Num uniques 859; 2023-04-09 09:55:20,017 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681034119997, max timestamp=1681034120015Num uniques 859; 2023-04-09 09:55:20,017 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-09 09:55:20,017 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681034119997, max timestamp=1681034120015Num uniques 859; 2023-04-09 09:55:20,017 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-09 09:55:20,017 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-09 09:55:20,036 DEBUG [Thread-2431] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:20,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:20,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:20,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:20,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:20,040 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:20,040 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:20,040 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:20,040 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:20,041 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:20,041 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:20,042 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:20,042 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:20,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:20,043 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,043 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:20,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:20,043 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:20,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:20,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:20,043 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:20,044 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:20,044 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:20,044 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:20,044 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:20,044 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:20,044 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:20,044 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:20,044 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:20,044 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. started... 2023-04-09 09:55:20,044 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. as already flushing 2023-04-09 09:55:20,044 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:20,044 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:20,044 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:20,044 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,044 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:20,044 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:20,047 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,047 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:20,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:20,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:20,047 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:20,047 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:20,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:20,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:20,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:20,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:20,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:20,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@82b46a4[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:20,050 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:20,050 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:20,052 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:20,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:20,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:20,052 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:20,052 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:20,052 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:20,052 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,052 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:20,053 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,053 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:20,053 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:20,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:20,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:20,053 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:20,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:20,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:20,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:20,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:20,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:20,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:20,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:20,055 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:20,055 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:20,055 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:20,055 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:20,055 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:20,056 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:20,056 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:20,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32856 deadline: 1681034180042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32788 deadline: 1681034180043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,056 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:20,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32882 deadline: 1681034180049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,057 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:20,057 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:20,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:20,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:20,057 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:20,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:20,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:20,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:20,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:20,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:20,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:20,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:20,058 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:20,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:20,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:20,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,059 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:20,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:20,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,062 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,062 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:20,062 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:20,062 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:20,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:20,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:20,063 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:20,063 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,063 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:20,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:20,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:20,063 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:20,063 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:20,063 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:20,063 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:20,063 DEBUG [Thread-2431] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:20,063 DEBUG [Thread-2431] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:20,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:20,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:20,162 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:20,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32856 deadline: 1681034180157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,162 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:20,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32882 deadline: 1681034180158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,163 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:20,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32788 deadline: 1681034180158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,163 DEBUG [Thread-2431] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:20,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:20,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:20,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:20,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:20,167 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:20,167 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:20,167 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:20,167 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:20,167 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:20,167 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:20,168 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:20,168 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:20,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:20,169 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:20,169 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:20,169 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:20,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:20,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:20,170 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:20,170 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:20,170 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:20,170 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:20,170 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:20,170 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:20,170 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:20,170 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:20,170 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:20,170 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. started... 2023-04-09 09:55:20,171 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. as already flushing 2023-04-09 09:55:20,171 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:20,171 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:20,171 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:20,171 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,171 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:20,171 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:20,173 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,173 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:20,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:20,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:20,173 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:20,173 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:20,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:20,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:20,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:20,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:20,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:20,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@666fee31[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:20,175 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:20,175 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:20,176 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:20,176 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,176 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:20,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:20,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:20,176 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:20,176 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:20,176 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:20,182 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:20,182 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,182 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:20,182 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:20,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:20,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:20,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:20,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:20,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:20,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:20,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:20,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:20,184 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:20,184 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:20,184 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:20,184 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:20,184 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:20,184 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:20,187 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:20,187 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:20,187 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:20,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:20,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:20,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:20,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:20,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:20,188 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:20,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:20,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:20,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:20,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:20,189 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:20,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:20,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:20,191 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:20,191 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:20,191 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:20,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:20,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:20,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:20,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:20,192 DEBUG [Thread-2431] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:20,192 DEBUG [Thread-2431] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:20,192 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:20,192 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,192 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:20,192 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:20,193 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,193 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:20,193 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:20,193 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:20,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:20,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:20,292 DEBUG [Thread-2431] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:20,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:20,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:20,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:20,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:20,297 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:20,297 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:20,297 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:20,298 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:20,298 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:20,298 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:20,299 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,299 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:20,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:20,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:20,299 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,299 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:20,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:20,300 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:20,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:20,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:20,300 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:20,301 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:20,301 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:20,301 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:20,301 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:20,301 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:20,301 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:20,301 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:20,301 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:20,301 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. started... 2023-04-09 09:55:20,301 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. as already flushing 2023-04-09 09:55:20,301 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:20,301 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:20,301 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:20,301 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,301 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:20,301 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:20,307 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:20,307 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:20,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:20,308 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:20,308 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:20,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:20,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:20,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:20,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:20,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:20,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d49501f[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:20,309 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:20,309 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:20,310 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:20,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:20,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:20,310 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:20,310 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,310 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:20,310 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:20,310 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:20,311 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:20,311 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,311 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:20,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:20,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:20,312 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:20,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:20,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:20,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:20,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:20,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:20,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:20,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:20,314 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:20,314 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:20,314 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:20,314 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:20,314 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:20,315 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:20,315 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:20,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:20,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:20,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:20,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:20,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:20,315 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:20,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:20,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:20,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:20,316 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:20,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:20,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:20,317 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:20,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:20,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:20,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,320 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,320 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:20,320 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:20,320 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:20,320 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:20,320 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:20,320 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:20,320 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:20,320 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:20,321 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:20,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:20,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:20,320 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:20,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:20,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:20,321 DEBUG [Thread-2431] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:20,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:20,321 DEBUG [Thread-2431] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:20,366 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:20,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32882 deadline: 1681034180363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,366 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:20,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32856 deadline: 1681034180364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,366 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:20,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32788 deadline: 1681034180364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,402 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=272 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/B/b1c649bb260d4bd6a8fa7949fe53b4f3 2023-04-09 09:55:20,421 DEBUG [Thread-2431] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:20,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:20,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:20,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:20,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:20,424 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:20,424 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:20,424 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:20,424 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:20,425 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:20,425 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:20,426 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:20,426 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:20,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:20,426 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,426 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:20,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:20,426 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:20,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:20,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:20,427 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:20,427 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:20,427 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:20,427 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:20,427 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:20,427 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:20,427 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:20,427 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:20,427 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:20,427 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. started... 2023-04-09 09:55:20,428 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. as already flushing 2023-04-09 09:55:20,428 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:20,428 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:20,428 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:20,428 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,428 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:20,428 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:20,429 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,429 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:20,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:20,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:20,429 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:20,429 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:20,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:20,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:20,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:20,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:20,431 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:20,431 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57f2f630[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:20,431 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:20,431 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:20,432 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:20,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:20,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:20,432 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:20,432 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:20,432 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:20,432 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,432 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:20,434 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,434 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:20,435 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:20,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:20,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:20,435 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:20,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:20,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:20,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:20,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:20,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:20,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:20,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:20,437 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:20,437 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:20,437 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:20,437 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:20,437 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:20,439 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:20,439 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:20,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:20,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:20,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:20,439 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:20,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:20,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:20,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:20,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:20,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:20,440 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:20,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:20,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:20,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:20,442 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:20,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:20,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,444 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:20,444 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:20,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:20,444 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:20,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:20,444 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,444 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:20,444 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:20,444 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,444 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:20,444 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:20,445 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:20,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:20,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:20,444 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:20,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:20,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:20,445 DEBUG [Thread-2431] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:20,445 DEBUG [Thread-2431] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:20,545 DEBUG [Thread-2431] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:20,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:20,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:20,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:20,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:20,548 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:20,548 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:20,548 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:20,548 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:20,549 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:20,549 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:20,550 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:20,550 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:20,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:20,550 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,550 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:20,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:20,550 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:20,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:20,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:20,551 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:20,551 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:20,551 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:20,551 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:20,551 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:20,551 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:20,551 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:20,551 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:20,552 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:20,552 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. started... 2023-04-09 09:55:20,552 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. as already flushing 2023-04-09 09:55:20,552 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:20,552 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:20,552 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:20,552 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,552 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:20,552 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:20,554 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:20,554 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:20,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:20,554 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:20,554 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:20,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:20,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:20,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:20,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:20,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:20,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53e940c[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:20,555 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:20,555 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:20,556 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:20,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:20,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:20,556 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:20,557 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:20,557 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:20,557 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,557 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:20,558 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,558 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:20,558 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:20,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:20,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:20,558 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:20,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:20,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:20,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:20,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:20,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:20,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:20,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:20,560 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:20,560 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:20,560 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:20,560 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:20,560 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:20,562 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:20,562 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:20,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:20,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:20,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:20,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:20,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:20,562 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:20,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:20,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:20,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:20,562 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:20,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:20,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:20,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:20,563 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:20,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:20,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,571 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:20,571 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:20,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:20,572 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:20,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:20,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:20,572 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:20,572 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:20,572 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:20,572 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,572 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:20,572 DEBUG [Thread-2431] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:20,572 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:20,572 DEBUG [Thread-2431] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:20,572 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:20,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:20,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:20,572 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:20,671 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:20,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32856 deadline: 1681034180668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,672 DEBUG [Thread-2431] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:20,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:20,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:20,675 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:20,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32788 deadline: 1681034180668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,675 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:20,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32882 deadline: 1681034180669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:20,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:20,680 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:20,680 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:20,680 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:20,680 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:20,682 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:20,682 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:20,683 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:20,683 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:20,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:20,684 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,684 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:20,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:20,684 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:20,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:20,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:20,685 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:20,685 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:20,685 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:20,685 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:20,685 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:20,685 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:20,685 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:20,685 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:20,685 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:20,685 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. started... 2023-04-09 09:55:20,685 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. as already flushing 2023-04-09 09:55:20,686 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:20,686 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:20,686 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:20,686 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,686 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:20,686 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:20,687 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:20,687 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:20,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:20,688 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:20,688 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:20,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:20,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:20,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:20,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:20,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:20,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1cb9ecfc[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:20,689 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:20,689 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:20,690 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:20,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:20,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:20,691 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:20,691 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:20,691 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,691 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:20,691 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:20,693 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:20,693 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:20,693 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:20,693 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:20,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:20,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:20,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:20,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:20,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:20,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:20,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:20,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:20,696 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:20,696 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:20,696 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:20,696 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:20,696 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:20,698 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:20,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:20,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:20,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:20,698 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:20,698 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:20,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:20,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:20,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:20,699 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:20,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:20,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:20,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:20,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:20,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:20,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,700 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:20,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:20,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,703 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:20,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:20,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:20,703 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:20,703 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:20,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:20,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:20,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:20,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:20,704 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:20,704 DEBUG [Thread-2431] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:20,704 DEBUG [Thread-2431] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:20,705 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,705 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:20,705 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:20,705 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,705 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:20,705 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:20,705 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:20,804 DEBUG [Thread-2431] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:20,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:20,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:20,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:20,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:20,808 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:20,808 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:20,808 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:20,808 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:20,812 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=272 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/C/9cf5c68d3fd7490493124dad1a99dbf3 2023-04-09 09:55:20,814 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:20,814 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:20,815 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:20,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:20,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:20,815 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:20,816 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,816 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:20,816 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:20,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:20,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:20,817 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:20,825 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:20,825 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:20,825 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:20,825 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:20,825 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:20,825 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/A/cd46d9c1278c4db296846d0bdadbbf98 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/cd46d9c1278c4db296846d0bdadbbf98 2023-04-09 09:55:20,828 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:20,828 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:20,828 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:20,828 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. started... 2023-04-09 09:55:20,828 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. as already flushing 2023-04-09 09:55:20,828 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:20,828 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:20,828 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:20,828 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,828 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:20,828 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:20,830 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/cd46d9c1278c4db296846d0bdadbbf98, entries=300, sequenceid=272, filesize=46.1 K 2023-04-09 09:55:20,830 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/B/b1c649bb260d4bd6a8fa7949fe53b4f3 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/b1c649bb260d4bd6a8fa7949fe53b4f3 2023-04-09 09:55:20,831 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,831 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:20,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:20,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:20,831 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:20,831 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:20,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:20,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:20,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:20,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:20,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:20,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b279edc[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:20,834 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:20,834 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:20,835 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:20,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:20,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:20,835 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:20,835 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:20,835 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:20,835 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,835 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:20,836 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:20,836 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:20,836 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:20,837 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:20,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:20,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:20,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:20,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:20,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:20,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:20,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:20,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:20,840 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:20,840 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:20,840 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:20,840 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:20,840 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:20,842 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:20,842 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:20,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:20,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:20,842 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:20,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:20,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:20,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:20,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:20,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:20,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:20,843 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:20,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:20,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:20,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:20,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,845 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:20,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:20,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,848 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,848 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:20,848 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:20,848 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:20,848 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:20,848 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:20,848 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:20,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:20,848 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:20,848 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,849 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:20,849 DEBUG [Thread-2431] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:20,849 DEBUG [Thread-2431] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:20,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:20,849 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:20,849 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:20,849 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:20,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:20,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:20,850 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/b1c649bb260d4bd6a8fa7949fe53b4f3, entries=150, sequenceid=272, filesize=11.9 K 2023-04-09 09:55:20,851 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/C/9cf5c68d3fd7490493124dad1a99dbf3 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/9cf5c68d3fd7490493124dad1a99dbf3 2023-04-09 09:55:20,857 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/9cf5c68d3fd7490493124dad1a99dbf3, entries=150, sequenceid=272, filesize=11.9 K 2023-04-09 09:55:20,858 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~146.34 KB/149856, currentSize=134.18 KB/137400 for bf27154b49f0831beaa90741ce575bc9 in 905ms, sequenceid=272, compaction requested=true 2023-04-09 09:55:20,858 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bf27154b49f0831beaa90741ce575bc9: 2023-04-09 09:55:20,858 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:55:20,858 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-09 09:55:20,858 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-09 09:55:20,858 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:55:20,859 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:55:20,859 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 92204 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-09 09:55:20,859 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34104 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-09 09:55:20,859 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1912): bf27154b49f0831beaa90741ce575bc9/A is initiating minor compaction (all files) 2023-04-09 09:55:20,860 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): bf27154b49f0831beaa90741ce575bc9/B is initiating minor compaction (all files) 2023-04-09 09:55:20,860 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bf27154b49f0831beaa90741ce575bc9/A in TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:20,860 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bf27154b49f0831beaa90741ce575bc9/B in TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:20,860 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/e0cbce6fc27c484bbc7a8b518327e484, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/a64933e6e15d43b39e74cfc55557e4ad, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/cd46d9c1278c4db296846d0bdadbbf98] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp, totalSize=90.0 K 2023-04-09 09:55:20,860 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/173860f44f104ca3a42cfc17f917511f, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/9169639d247a48a981b1b7a471bdca44, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/b1c649bb260d4bd6a8fa7949fe53b4f3] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp, totalSize=33.3 K 2023-04-09 09:55:20,860 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting e0cbce6fc27c484bbc7a8b518327e484, keycount=150, bloomtype=ROW, size=25.5 K, encoding=NONE, compression=NONE, seqNum=151, earliestPutTs=1681034117260 2023-04-09 09:55:20,860 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 173860f44f104ca3a42cfc17f917511f, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=151, earliestPutTs=1681034117260 2023-04-09 09:55:20,860 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting a64933e6e15d43b39e74cfc55557e4ad, keycount=100, bloomtype=ROW, size=18.5 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1681034118421 2023-04-09 09:55:20,861 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 9169639d247a48a981b1b7a471bdca44, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1681034118421 2023-04-09 09:55:20,861 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting cd46d9c1278c4db296846d0bdadbbf98, keycount=300, bloomtype=ROW, size=46.1 K, encoding=NONE, compression=NONE, seqNum=272, earliestPutTs=1681034119838 2023-04-09 09:55:20,863 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting b1c649bb260d4bd6a8fa7949fe53b4f3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=272, earliestPutTs=1681034119944 2023-04-09 09:55:20,881 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): bf27154b49f0831beaa90741ce575bc9#B#compaction#358 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:55:20,889 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] throttle.PressureAwareThroughputController(145): bf27154b49f0831beaa90741ce575bc9#A#compaction#357 average throughput is 1.97 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:55:20,900 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/B/499248fcbec64e408ac9c394328633a0 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/499248fcbec64e408ac9c394328633a0 2023-04-09 09:55:20,904 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bf27154b49f0831beaa90741ce575bc9/B of bf27154b49f0831beaa90741ce575bc9 into 499248fcbec64e408ac9c394328633a0(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:55:20,904 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bf27154b49f0831beaa90741ce575bc9: 2023-04-09 09:55:20,904 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9., storeName=bf27154b49f0831beaa90741ce575bc9/B, priority=13, startTime=1681034120858; duration=0sec 2023-04-09 09:55:20,904 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:55:20,904 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-09 09:55:20,905 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34104 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-09 09:55:20,905 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): bf27154b49f0831beaa90741ce575bc9/C is initiating minor compaction (all files) 2023-04-09 09:55:20,905 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bf27154b49f0831beaa90741ce575bc9/C in TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:20,905 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/b7d65a78164240969fdafee199635db2, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/27458d76c0034be3bc60a79ca3306061, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/9cf5c68d3fd7490493124dad1a99dbf3] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp, totalSize=33.3 K 2023-04-09 09:55:20,905 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting b7d65a78164240969fdafee199635db2, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=151, earliestPutTs=1681034117260 2023-04-09 09:55:20,906 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 27458d76c0034be3bc60a79ca3306061, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1681034118421 2023-04-09 09:55:20,906 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 9cf5c68d3fd7490493124dad1a99dbf3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=272, earliestPutTs=1681034119944 2023-04-09 09:55:20,910 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): bf27154b49f0831beaa90741ce575bc9#C#compaction#359 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:55:20,919 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/C/cef0e2878c60471bb3fc23042940ba10 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/cef0e2878c60471bb3fc23042940ba10 2023-04-09 09:55:20,924 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bf27154b49f0831beaa90741ce575bc9/C of bf27154b49f0831beaa90741ce575bc9 into cef0e2878c60471bb3fc23042940ba10(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:55:20,924 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bf27154b49f0831beaa90741ce575bc9: 2023-04-09 09:55:20,924 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9., storeName=bf27154b49f0831beaa90741ce575bc9/C, priority=13, startTime=1681034120859; duration=0sec 2023-04-09 09:55:20,924 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:55:20,949 DEBUG [Thread-2431] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:20,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:20,951 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:20,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:20,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:20,952 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:20,952 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:20,952 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:20,953 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:20,953 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:20,953 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:20,954 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,954 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:20,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:20,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:20,954 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:20,954 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:20,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:20,955 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:20,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:20,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:20,955 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:20,955 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:20,955 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:20,956 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:20,956 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:20,956 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:20,956 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:20,956 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:20,956 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:20,956 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. started... 2023-04-09 09:55:20,956 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bf27154b49f0831beaa90741ce575bc9 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-04-09 09:55:20,956 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf27154b49f0831beaa90741ce575bc9, store=A 2023-04-09 09:55:20,956 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:55:20,956 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf27154b49f0831beaa90741ce575bc9, store=B 2023-04-09 09:55:20,957 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:55:20,957 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf27154b49f0831beaa90741ce575bc9, store=C 2023-04-09 09:55:20,957 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:55:20,975 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304096ec4f86818a04afe97cb01151a51c383 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304096ec4f86818a04afe97cb01151a51c383 2023-04-09 09:55:20,989 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=309, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/A/73d8c0c0dddd42cba1445198b67b32ec 2023-04-09 09:55:21,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on bf27154b49f0831beaa90741ce575bc9 2023-04-09 09:55:21,180 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. as already flushing 2023-04-09 09:55:21,191 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:21,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32882 deadline: 1681034181187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,192 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:21,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32788 deadline: 1681034181189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,195 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:21,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32856 deadline: 1681034181190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,296 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:21,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32882 deadline: 1681034181292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,297 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:21,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32788 deadline: 1681034181293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,298 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:21,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32856 deadline: 1681034181296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,404 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=309 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/B/00767cdbf95748e8b61ccc0f2b3e3a20 2023-04-09 09:55:21,416 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=309 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/C/3ee5161b43d246ebbf00ff8bf515831f 2023-04-09 09:55:21,421 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/A/73d8c0c0dddd42cba1445198b67b32ec as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/73d8c0c0dddd42cba1445198b67b32ec 2023-04-09 09:55:21,427 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/73d8c0c0dddd42cba1445198b67b32ec, entries=150, sequenceid=309, filesize=25.4 K 2023-04-09 09:55:21,428 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/B/00767cdbf95748e8b61ccc0f2b3e3a20 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/00767cdbf95748e8b61ccc0f2b3e3a20 2023-04-09 09:55:21,438 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/00767cdbf95748e8b61ccc0f2b3e3a20, entries=150, sequenceid=309, filesize=11.9 K 2023-04-09 09:55:21,438 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/C/3ee5161b43d246ebbf00ff8bf515831f as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/3ee5161b43d246ebbf00ff8bf515831f 2023-04-09 09:55:21,445 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/3ee5161b43d246ebbf00ff8bf515831f, entries=150, sequenceid=309, filesize=11.9 K 2023-04-09 09:55:21,446 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=46.96 KB/48090 for bf27154b49f0831beaa90741ce575bc9 in 490ms, sequenceid=309, compaction requested=false 2023-04-09 09:55:21,447 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bf27154b49f0831beaa90741ce575bc9: 2023-04-09 09:55:21,447 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:21,447 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:21,447 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:21,447 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,447 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:21,447 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:21,449 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:21,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:21,449 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:21,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:21,449 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:21,449 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:21,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:21,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:21,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:21,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:21,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b24b2fe[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:21,451 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:21,451 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:21,452 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:21,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:21,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:21,452 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:21,452 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:21,453 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:21,453 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,453 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:21,454 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:21,454 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:21,454 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:21,454 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:21,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:21,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:21,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:21,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:21,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:21,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:21,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:21,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:21,457 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:21,457 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:21,457 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:21,457 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:21,457 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:21,459 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:21,459 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:21,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:21,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:21,459 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:21,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:21,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:21,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:21,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:21,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:21,460 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:21,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:21,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:21,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:21,463 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:21,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:21,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:21,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,469 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,470 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:21,470 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:21,470 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:21,470 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:21,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:21,470 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:21,470 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:21,470 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:21,470 DEBUG [Thread-2431] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:21,470 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:21,470 DEBUG [Thread-2431] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:21,472 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:21,472 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:21,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:21,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:21,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:21,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:21,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on bf27154b49f0831beaa90741ce575bc9 2023-04-09 09:55:21,502 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bf27154b49f0831beaa90741ce575bc9 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-09 09:55:21,502 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf27154b49f0831beaa90741ce575bc9, store=A 2023-04-09 09:55:21,503 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:21,503 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf27154b49f0831beaa90741ce575bc9, store=B 2023-04-09 09:55:21,503 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:21,503 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf27154b49f0831beaa90741ce575bc9, store=C 2023-04-09 09:55:21,503 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:21,517 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230409187b9c6d248c4f06b6da43c534690e1a to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409187b9c6d248c4f06b6da43c534690e1a 2023-04-09 09:55:21,563 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:21,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32882 deadline: 1681034181556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,564 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:21,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32856 deadline: 1681034181557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,564 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:21,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32788 deadline: 1681034181563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,570 DEBUG [Thread-2431] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:21,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:21,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:21,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:21,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:21,586 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:21,586 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:21,586 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:21,586 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:21,587 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:21,587 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:21,588 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:21,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:21,588 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:21,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:21,589 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,589 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:21,589 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:21,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:21,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:21,589 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:21,593 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:21,593 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:21,593 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:21,593 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:21,593 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:21,596 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:21,596 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:21,596 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:21,596 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. started... 2023-04-09 09:55:21,596 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. as already flushing 2023-04-09 09:55:21,596 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:21,596 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:21,596 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:21,596 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,597 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:21,597 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:21,598 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:21,598 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:21,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:21,598 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:21,598 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:21,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:21,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:21,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:21,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:21,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:21,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a4bc962[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:21,600 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:21,600 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:21,601 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:21,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:21,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:21,602 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:21,602 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:21,602 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:21,602 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,602 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:21,604 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:21,604 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,604 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:21,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:21,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:21,604 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:21,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:21,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:21,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:21,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:21,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:21,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:21,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:21,607 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:21,607 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:21,607 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:21,607 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:21,607 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:21,609 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:21,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:21,609 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:21,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:21,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:21,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:21,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:21,609 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:21,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:21,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:21,610 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:21,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:21,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:21,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:21,611 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:21,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:21,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:21,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,614 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:21,614 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:21,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:21,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:21,615 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:21,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:21,615 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:21,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:21,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:21,615 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:21,615 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:21,616 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:21,615 DEBUG [Thread-2431] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:21,616 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,616 DEBUG [Thread-2431] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:21,616 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:21,616 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:21,616 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:21,672 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:21,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32882 deadline: 1681034181665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,673 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:21,673 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:21,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32856 deadline: 1681034181665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32788 deadline: 1681034181665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,716 DEBUG [Thread-2431] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:21,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:21,718 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/A/77efd3581c054e84a8328fe75cb55b1c as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/77efd3581c054e84a8328fe75cb55b1c 2023-04-09 09:55:21,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:21,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:21,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:21,721 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:21,721 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:21,721 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:21,721 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:21,722 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:21,722 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:21,723 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bf27154b49f0831beaa90741ce575bc9/A of bf27154b49f0831beaa90741ce575bc9 into 77efd3581c054e84a8328fe75cb55b1c(size=25.8 K), total size for store is 51.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:55:21,723 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bf27154b49f0831beaa90741ce575bc9: 2023-04-09 09:55:21,723 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9., storeName=bf27154b49f0831beaa90741ce575bc9/A, priority=13, startTime=1681034120858; duration=0sec 2023-04-09 09:55:21,723 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:55:21,729 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:21,729 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:21,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:21,729 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,729 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:21,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:21,730 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:21,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:21,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:21,730 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:21,730 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:21,730 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:21,730 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:21,730 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:21,730 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:21,730 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:21,730 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:21,730 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:21,730 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. started... 2023-04-09 09:55:21,731 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. as already flushing 2023-04-09 09:55:21,731 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:21,731 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:21,731 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:21,731 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,731 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:21,731 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:21,732 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,733 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:21,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:21,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:21,733 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:21,733 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:21,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:21,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:21,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:21,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:21,735 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:21,735 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:21,735 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29c614db[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:21,735 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:21,736 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:21,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:21,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:21,736 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:21,736 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:21,736 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,736 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:21,736 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:21,738 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:21,738 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:21,738 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,738 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:21,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:21,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:21,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:21,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:21,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:21,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:21,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:21,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:21,743 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:21,743 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:21,743 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:21,743 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:21,743 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:21,743 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:21,746 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,746 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:21,746 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:21,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:21,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:21,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:21,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:21,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:21,747 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,748 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:21,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:21,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:21,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:21,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:21,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:21,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:21,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:21,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:21,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:21,751 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:21,751 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:21,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:21,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:21,751 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:21,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:21,751 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:21,751 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:21,752 DEBUG [Thread-2431] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:21,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:21,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:21,752 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:21,752 DEBUG [Thread-2431] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:21,752 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:21,753 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,753 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:21,753 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:21,753 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:21,853 DEBUG [Thread-2431] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:21,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:21,855 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:21,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:21,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:21,856 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:21,856 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:21,856 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:21,856 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:21,856 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:21,856 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:21,857 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,857 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:21,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:21,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:21,858 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,858 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:21,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:21,858 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:21,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:21,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:21,859 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:21,859 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:21,859 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:21,859 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:21,859 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:21,859 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:21,859 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:21,859 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:21,859 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:21,859 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. started... 2023-04-09 09:55:21,859 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. as already flushing 2023-04-09 09:55:21,859 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:21,859 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:21,860 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:21,860 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,860 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:21,860 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:21,861 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,861 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:21,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:21,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:21,861 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:21,861 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:21,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:21,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:21,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:21,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:21,863 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:21,863 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ee0be41[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:21,863 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:21,863 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:21,865 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:21,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:21,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:21,865 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:21,865 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:21,865 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:21,865 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,865 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:21,866 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,866 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:21,866 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:21,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:21,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:21,866 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:21,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:21,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:21,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:21,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:21,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:21,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:21,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:21,869 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:21,869 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:21,869 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:21,869 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:21,869 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:21,870 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:21,870 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:21,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:21,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:21,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:21,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:21,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:21,870 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:21,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:21,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:21,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:21,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:21,871 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:21,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:21,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:21,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,872 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:21,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:21,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,874 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,874 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:21,874 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:21,874 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:21,874 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:21,874 DEBUG [Thread-2431] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:21,874 DEBUG [Thread-2431] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:21,874 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:21,875 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,875 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:21,875 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:21,875 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:21,874 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:21,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:21,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:21,874 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:21,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:21,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:21,877 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:21,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32788 deadline: 1681034181874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,877 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:21,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32882 deadline: 1681034181875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,877 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:21,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32856 deadline: 1681034181875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,923 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=321, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/A/e5e307222e684b94b625fc9b70cb0bd7 2023-04-09 09:55:21,975 DEBUG [Thread-2431] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:21,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:21,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:21,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:21,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:21,978 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:21,978 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:21,978 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:21,978 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:21,979 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:21,979 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:21,983 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:21,983 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:21,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:21,984 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,984 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:21,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:21,984 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:21,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:21,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:21,985 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:21,985 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:21,985 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:21,985 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:21,985 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:21,985 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:21,985 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:21,985 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:21,985 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:21,985 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. started... 2023-04-09 09:55:21,985 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. as already flushing 2023-04-09 09:55:21,985 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:21,985 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:21,985 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:21,985 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,985 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:21,986 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:21,987 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:21,987 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:21,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:21,987 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:21,987 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:21,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:21,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:21,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:21,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:21,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:21,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70fd5421[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:21,988 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:21,989 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:21,989 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:21,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:21,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:21,990 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:21,990 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:21,990 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:21,990 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,990 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:21,991 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,991 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:21,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:21,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:21,991 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:21,991 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:21,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:21,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:21,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:21,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:21,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:21,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:21,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:21,993 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:21,993 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:21,993 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:21,993 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:21,993 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:21,996 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:21,996 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:21,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:21,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:21,996 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:21,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:21,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:21,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:21,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:21,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:21,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:21,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:21,996 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:21,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:21,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:21,998 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:21,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:21,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:21,999 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:21,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:21,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:21,999 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,000 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:22,000 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:22,000 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:22,000 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:22,000 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:22,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:22,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:22,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:22,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:22,000 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,000 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:22,000 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:22,000 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:22,000 DEBUG [Thread-2431] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:22,000 DEBUG [Thread-2431] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:22,100 DEBUG [Thread-2431] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:22,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:22,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:22,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:22,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:22,104 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:22,104 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:22,104 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:22,104 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:22,104 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:22,104 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:22,105 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,105 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:22,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:22,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:22,106 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,106 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:22,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:22,106 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:22,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:22,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:22,107 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:22,107 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:22,107 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:22,107 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:22,107 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:22,107 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:22,108 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:22,108 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:22,108 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:22,108 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. started... 2023-04-09 09:55:22,108 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. as already flushing 2023-04-09 09:55:22,108 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:22,108 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:22,108 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:22,108 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,108 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:22,108 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:22,109 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:22,109 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:22,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:22,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:22,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:22,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:22,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:22,111 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:22,111 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:22,112 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:22,112 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54d98385[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:22,112 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:22,112 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:22,114 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:22,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:22,114 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,114 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:22,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:22,114 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:22,114 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:22,114 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:22,116 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:22,116 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:22,116 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,116 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:22,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:22,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:22,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:22,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:22,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:22,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:22,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:22,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:22,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:22,119 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:22,119 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:22,119 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:22,119 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:22,119 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:22,121 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:22,121 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:22,121 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:22,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:22,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:22,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:22,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:22,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:22,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:22,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:22,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:22,122 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:22,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:22,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:22,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:22,123 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:22,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:22,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,125 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:22,125 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:22,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:22,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:22,126 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:22,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:22,126 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:22,126 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:22,126 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,126 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:22,126 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:22,126 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:22,126 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:22,126 DEBUG [Thread-2431] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:22,126 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:22,126 DEBUG [Thread-2431] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:22,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:22,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:22,181 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:22,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32788 deadline: 1681034182179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,181 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:22,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32856 deadline: 1681034182180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,181 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:22,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32882 deadline: 1681034182180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,226 DEBUG [Thread-2431] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:22,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:22,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:22,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:22,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:22,229 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:22,229 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:22,229 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:22,229 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:22,230 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:22,230 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:22,231 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:22,231 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:22,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:22,231 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,231 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:22,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:22,232 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:22,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:22,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:22,233 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:22,233 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:22,233 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:22,233 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:22,233 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:22,233 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:22,235 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:22,235 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:22,235 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:22,235 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. started... 2023-04-09 09:55:22,235 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. as already flushing 2023-04-09 09:55:22,236 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:22,236 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:22,236 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:22,236 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,236 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:22,236 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:22,237 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,237 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:22,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:22,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:22,237 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:22,237 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:22,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:22,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:22,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:22,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:22,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:22,239 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:22,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4da02509[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:22,239 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:22,240 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:22,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:22,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:22,240 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:22,240 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,240 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:22,240 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:22,240 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:22,243 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:22,243 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:22,243 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,243 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:22,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:22,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:22,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:22,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:22,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:22,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:22,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:22,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:22,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:22,245 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:22,245 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:22,245 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:22,245 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:22,245 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:22,246 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:22,246 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:22,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:22,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:22,246 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:22,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:22,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:22,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:22,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:22,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:22,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:22,247 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:22,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:22,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:22,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:22,248 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:22,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:22,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,250 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:22,250 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,250 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:22,250 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:22,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:22,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:22,250 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:22,251 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:22,250 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:22,251 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:22,251 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,251 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:22,251 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:22,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:22,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:22,251 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:22,251 DEBUG [Thread-2431] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:22,251 DEBUG [Thread-2431] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:22,333 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=321 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/B/b299072101ea486a9e0e7220c9482f9e 2023-04-09 09:55:22,351 DEBUG [Thread-2431] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:22,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:22,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:22,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:22,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:22,356 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:22,356 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:22,356 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:22,356 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:22,356 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:22,356 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:22,357 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,357 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:22,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:22,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:22,358 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,358 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:22,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:22,358 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:22,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:22,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:22,359 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:22,359 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:22,359 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:22,359 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:22,359 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:22,359 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:22,359 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:22,359 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:22,359 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:22,359 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. started... 2023-04-09 09:55:22,359 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. as already flushing 2023-04-09 09:55:22,361 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:22,361 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:22,361 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:22,361 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,361 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:22,361 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:22,366 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,366 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:22,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:22,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:22,367 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:22,367 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:22,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:22,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:22,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:22,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:22,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:22,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1669884a[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:22,368 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:22,368 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:22,369 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:22,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:22,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:22,369 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:22,369 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:22,370 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:22,370 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,370 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:22,371 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:22,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:22,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:22,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:22,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:22,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:22,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:22,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:22,374 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:22,374 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:22,374 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:22,374 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:22,374 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:22,374 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:22,375 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:22,375 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:22,375 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:22,377 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:22,377 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:22,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:22,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:22,378 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,378 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:22,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:22,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:22,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:22,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:22,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:22,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:22,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:22,379 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:22,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:22,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:22,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:22,381 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,381 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:22,382 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:22,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:22,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:22,382 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:22,382 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,382 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:22,382 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:22,382 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:22,382 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:22,382 DEBUG [Thread-2431] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:22,382 DEBUG [Thread-2431] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:22,382 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:22,382 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:22,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:22,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:22,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:22,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:22,482 DEBUG [Thread-2431] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:22,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:22,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:22,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:22,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:22,486 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:22,486 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:22,486 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:22,486 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:22,486 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:22,486 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:22,487 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:22,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32772 deadline: 1681034182484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,488 DEBUG [Thread-2425] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8179 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9., hostname=jenkins-hbase4.apache.org,42467,1681034019544, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-09 09:55:22,489 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:22,489 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:22,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:22,489 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,489 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:22,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:22,489 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:22,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:22,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:22,490 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:22,490 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:22,490 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:22,490 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:22,490 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:22,490 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:22,490 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:22,490 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:22,490 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:22,490 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. started... 2023-04-09 09:55:22,490 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. as already flushing 2023-04-09 09:55:22,490 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:22,491 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:22,491 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:22,491 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,491 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:22,491 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:22,492 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,492 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:22,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:22,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:22,492 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:22,492 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:22,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:22,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:22,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:22,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:22,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:22,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@caf9afa[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:22,494 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:22,494 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:22,496 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:22,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:22,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:22,496 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:22,496 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:22,496 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,496 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:22,496 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:22,497 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:22,497 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:22,497 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,497 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:22,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:22,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:22,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:22,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:22,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:22,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:22,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:22,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:22,499 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:22,499 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:22,499 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:22,499 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:22,499 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:22,499 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:22,501 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:22,501 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:22,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:22,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:22,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:22,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:22,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:22,501 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:22,501 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:22,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:22,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32870 deadline: 1681034182497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,501 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,503 DEBUG [Thread-2427] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8194 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9., hostname=jenkins-hbase4.apache.org,42467,1681034019544, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-09 09:55:22,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:22,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:22,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:22,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:22,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:22,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:22,504 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:22,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:22,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,506 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,506 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:22,506 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:22,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:22,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:22,506 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:22,506 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:22,506 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:22,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:22,506 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:22,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:22,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:22,506 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:22,507 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:22,507 DEBUG [Thread-2431] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:22,507 DEBUG [Thread-2431] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:22,507 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:22,607 DEBUG [Thread-2431] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:22,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:22,609 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:22,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:22,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:22,611 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:22,611 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:22,611 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:22,611 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:22,611 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:22,611 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:22,613 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:22,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:22,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:22,613 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:22,614 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,614 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:22,614 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:22,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:22,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:22,615 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:22,615 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:22,615 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:22,615 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:22,615 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:22,615 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:22,615 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:22,615 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:22,615 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:22,615 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. started... 2023-04-09 09:55:22,615 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. as already flushing 2023-04-09 09:55:22,615 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:22,615 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:22,615 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:22,615 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,616 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:22,616 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:22,617 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:22,617 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:22,617 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:22,618 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:22,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:22,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:22,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:22,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:22,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:22,621 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:22,621 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14f2f541[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:22,621 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:22,621 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:22,622 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:22,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:22,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:22,622 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:22,622 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:22,622 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,622 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:22,622 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:22,623 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,623 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:22,624 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:22,624 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:22,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:22,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:22,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:22,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:22,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:22,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:22,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:22,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:22,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:22,630 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:22,630 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:22,630 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:22,630 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:22,630 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:22,633 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:22,633 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:22,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:22,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:22,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:22,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:22,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:22,633 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:22,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:22,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:22,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:22,634 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:22,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:22,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:22,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:22,635 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:22,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:22,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,637 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,637 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:22,637 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:22,637 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,637 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:22,637 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:22,637 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:22,637 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:22,637 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:22,637 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:22,637 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:22,637 DEBUG [Thread-2431] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:22,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:22,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:22,637 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:22,637 DEBUG [Thread-2431] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:22,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:22,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:22,687 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:22,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32856 deadline: 1681034182684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,688 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:22,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32788 deadline: 1681034182686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,690 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:22,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32882 deadline: 1681034182686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,737 DEBUG [Thread-2431] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:22,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:22,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:22,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:22,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:22,741 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:22,741 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:22,741 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:22,741 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:22,742 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:22,742 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:22,743 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:22,743 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:22,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:22,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:22,743 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,743 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:22,743 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:22,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:22,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:22,744 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:22,744 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:22,744 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:22,744 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:22,744 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:22,744 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:22,744 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:22,744 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=321 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/C/6c68f163b0764b2ebc518f1eb148039b 2023-04-09 09:55:22,744 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:22,745 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. started... 2023-04-09 09:55:22,744 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:22,745 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. as already flushing 2023-04-09 09:55:22,745 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:22,745 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:22,745 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:22,745 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,745 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:22,745 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:22,746 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,746 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:22,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:22,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:22,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:22,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:22,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:22,748 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:22,748 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:22,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:22,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:22,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@583602bb[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:22,749 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:22,749 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:22,750 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/A/e5e307222e684b94b625fc9b70cb0bd7 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/e5e307222e684b94b625fc9b70cb0bd7 2023-04-09 09:55:22,751 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:22,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:22,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:22,752 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:22,752 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:22,752 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:22,752 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,752 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:22,754 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/e5e307222e684b94b625fc9b70cb0bd7, entries=200, sequenceid=321, filesize=32.3 K 2023-04-09 09:55:22,755 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/B/b299072101ea486a9e0e7220c9482f9e as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/b299072101ea486a9e0e7220c9482f9e 2023-04-09 09:55:22,755 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:22,755 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,755 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:22,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:22,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:22,755 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:22,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:22,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:22,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:22,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:22,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:22,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:22,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:22,758 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:22,758 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:22,758 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:22,758 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:22,758 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:22,760 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/b299072101ea486a9e0e7220c9482f9e, entries=150, sequenceid=321, filesize=11.9 K 2023-04-09 09:55:22,761 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/C/6c68f163b0764b2ebc518f1eb148039b as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/6c68f163b0764b2ebc518f1eb148039b 2023-04-09 09:55:22,761 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:22,761 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:22,761 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:22,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:22,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:22,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:22,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:22,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:22,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:22,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:22,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:22,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:22,762 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:22,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:22,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:22,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,764 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:22,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:22,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,766 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/6c68f163b0764b2ebc518f1eb148039b, entries=150, sequenceid=321, filesize=11.9 K 2023-04-09 09:55:22,766 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:22,766 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:22,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:22,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:22,767 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for bf27154b49f0831beaa90741ce575bc9 in 1265ms, sequenceid=321, compaction requested=true 2023-04-09 09:55:22,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:22,767 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bf27154b49f0831beaa90741ce575bc9: 2023-04-09 09:55:22,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:22,767 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:22,767 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-09 09:55:22,767 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,767 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:55:22,767 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:22,767 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:55:22,767 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-09 09:55:22,767 DEBUG [Thread-2431] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:22,767 DEBUG [Thread-2431] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:22,767 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:55:22,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:22,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:22,767 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:22,768 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,768 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:22,768 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:22,768 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:22,776 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85520 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-09 09:55:22,776 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1912): bf27154b49f0831beaa90741ce575bc9/A is initiating minor compaction (all files) 2023-04-09 09:55:22,776 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bf27154b49f0831beaa90741ce575bc9/A in TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:22,776 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36900 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-09 09:55:22,776 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/77efd3581c054e84a8328fe75cb55b1c, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/73d8c0c0dddd42cba1445198b67b32ec, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/e5e307222e684b94b625fc9b70cb0bd7] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp, totalSize=83.5 K 2023-04-09 09:55:22,776 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): bf27154b49f0831beaa90741ce575bc9/B is initiating minor compaction (all files) 2023-04-09 09:55:22,776 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bf27154b49f0831beaa90741ce575bc9/B in TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:22,776 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/499248fcbec64e408ac9c394328633a0, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/00767cdbf95748e8b61ccc0f2b3e3a20, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/b299072101ea486a9e0e7220c9482f9e] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp, totalSize=36.0 K 2023-04-09 09:55:22,777 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 77efd3581c054e84a8328fe75cb55b1c, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=272, earliestPutTs=1681034119944 2023-04-09 09:55:22,778 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 499248fcbec64e408ac9c394328633a0, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=272, earliestPutTs=1681034119944 2023-04-09 09:55:22,778 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 73d8c0c0dddd42cba1445198b67b32ec, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=309, earliestPutTs=1681034120039 2023-04-09 09:55:22,778 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 00767cdbf95748e8b61ccc0f2b3e3a20, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=309, earliestPutTs=1681034120039 2023-04-09 09:55:22,778 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting e5e307222e684b94b625fc9b70cb0bd7, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=321, earliestPutTs=1681034121179 2023-04-09 09:55:22,778 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting b299072101ea486a9e0e7220c9482f9e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=321, earliestPutTs=1681034121179 2023-04-09 09:55:22,788 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): bf27154b49f0831beaa90741ce575bc9#B#compaction#367 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:55:22,807 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] throttle.PressureAwareThroughputController(145): bf27154b49f0831beaa90741ce575bc9#A#compaction#366 average throughput is 0.99 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:55:22,817 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/B/f0d4f99efe0449e58ab816307ca3b37c as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/f0d4f99efe0449e58ab816307ca3b37c 2023-04-09 09:55:22,826 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bf27154b49f0831beaa90741ce575bc9/B of bf27154b49f0831beaa90741ce575bc9 into f0d4f99efe0449e58ab816307ca3b37c(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:55:22,826 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bf27154b49f0831beaa90741ce575bc9: 2023-04-09 09:55:22,826 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9., storeName=bf27154b49f0831beaa90741ce575bc9/B, priority=13, startTime=1681034122767; duration=0sec 2023-04-09 09:55:22,826 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:55:22,826 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-09 09:55:22,828 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36900 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-09 09:55:22,828 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): bf27154b49f0831beaa90741ce575bc9/C is initiating minor compaction (all files) 2023-04-09 09:55:22,828 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bf27154b49f0831beaa90741ce575bc9/C in TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:22,828 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/cef0e2878c60471bb3fc23042940ba10, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/3ee5161b43d246ebbf00ff8bf515831f, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/6c68f163b0764b2ebc518f1eb148039b] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp, totalSize=36.0 K 2023-04-09 09:55:22,828 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting cef0e2878c60471bb3fc23042940ba10, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=272, earliestPutTs=1681034119944 2023-04-09 09:55:22,829 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 3ee5161b43d246ebbf00ff8bf515831f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=309, earliestPutTs=1681034120039 2023-04-09 09:55:22,829 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 6c68f163b0764b2ebc518f1eb148039b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=321, earliestPutTs=1681034121179 2023-04-09 09:55:22,834 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): bf27154b49f0831beaa90741ce575bc9#C#compaction#368 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:55:22,868 DEBUG [Thread-2431] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:22,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:22,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:22,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:22,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:22,872 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:22,872 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:22,872 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:22,872 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:22,872 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:22,872 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:22,875 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,875 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:22,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:22,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:22,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:22,875 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:22,875 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:22,875 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:22,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:22,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:22,876 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:22,876 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:22,876 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:22,876 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:22,876 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:22,878 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:22,878 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:22,878 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:22,878 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. started... 2023-04-09 09:55:22,878 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bf27154b49f0831beaa90741ce575bc9 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-09 09:55:22,878 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf27154b49f0831beaa90741ce575bc9, store=A 2023-04-09 09:55:22,878 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:22,878 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf27154b49f0831beaa90741ce575bc9, store=B 2023-04-09 09:55:22,878 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:22,878 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf27154b49f0831beaa90741ce575bc9, store=C 2023-04-09 09:55:22,879 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:23,248 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/C/b9cff78f8f5f42a08da671626df1c61e as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/b9cff78f8f5f42a08da671626df1c61e 2023-04-09 09:55:23,255 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bf27154b49f0831beaa90741ce575bc9/C of bf27154b49f0831beaa90741ce575bc9 into b9cff78f8f5f42a08da671626df1c61e(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:55:23,255 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bf27154b49f0831beaa90741ce575bc9: 2023-04-09 09:55:23,255 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9., storeName=bf27154b49f0831beaa90741ce575bc9/C, priority=13, startTime=1681034122767; duration=0sec 2023-04-09 09:55:23,256 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:55:23,295 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040940e42f6a39484955bd61ead508186db8 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040940e42f6a39484955bd61ead508186db8 2023-04-09 09:55:23,638 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/A/7512c91429784267933243c027150373 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/7512c91429784267933243c027150373 2023-04-09 09:55:23,642 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bf27154b49f0831beaa90741ce575bc9/A of bf27154b49f0831beaa90741ce575bc9 into 7512c91429784267933243c027150373(size=25.9 K), total size for store is 25.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:55:23,642 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bf27154b49f0831beaa90741ce575bc9: 2023-04-09 09:55:23,642 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9., storeName=bf27154b49f0831beaa90741ce575bc9/A, priority=13, startTime=1681034122767; duration=0sec 2023-04-09 09:55:23,642 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:55:23,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on bf27154b49f0831beaa90741ce575bc9 2023-04-09 09:55:23,692 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. as already flushing 2023-04-09 09:55:23,709 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:23,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32856 deadline: 1681034183704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:23,713 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:23,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32882 deadline: 1681034183708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:23,713 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:23,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32788 deadline: 1681034183708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:23,714 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=343, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/A/c25885a6d5d14e2d894d1c6a7738bec8 2023-04-09 09:55:23,813 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:23,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32856 deadline: 1681034183810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:23,816 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:23,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32882 deadline: 1681034183814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:23,816 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:23,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32788 deadline: 1681034183814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:24,018 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:24,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32856 deadline: 1681034184015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:24,020 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:24,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32882 deadline: 1681034184017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:24,020 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:24,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32788 deadline: 1681034184018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:24,128 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=343 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/B/75b37fef0ce8492180570b1a1cdd336d 2023-04-09 09:55:24,322 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:24,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32856 deadline: 1681034184319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:24,326 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:24,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32882 deadline: 1681034184323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:24,327 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:24,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32788 deadline: 1681034184323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:24,538 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=343 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/C/9bc340e4dd5c4eb7a24905b6dd04a105 2023-04-09 09:55:24,542 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/A/c25885a6d5d14e2d894d1c6a7738bec8 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/c25885a6d5d14e2d894d1c6a7738bec8 2023-04-09 09:55:24,545 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/c25885a6d5d14e2d894d1c6a7738bec8, entries=150, sequenceid=343, filesize=25.4 K 2023-04-09 09:55:24,546 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/B/75b37fef0ce8492180570b1a1cdd336d as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/75b37fef0ce8492180570b1a1cdd336d 2023-04-09 09:55:24,568 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/75b37fef0ce8492180570b1a1cdd336d, entries=150, sequenceid=343, filesize=11.9 K 2023-04-09 09:55:24,569 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/C/9bc340e4dd5c4eb7a24905b6dd04a105 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/9bc340e4dd5c4eb7a24905b6dd04a105 2023-04-09 09:55:24,587 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/9bc340e4dd5c4eb7a24905b6dd04a105, entries=150, sequenceid=343, filesize=11.9 K 2023-04-09 09:55:24,587 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for bf27154b49f0831beaa90741ce575bc9 in 1709ms, sequenceid=343, compaction requested=false 2023-04-09 09:55:24,588 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bf27154b49f0831beaa90741ce575bc9: 2023-04-09 09:55:24,588 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:24,588 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:24,588 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:24,588 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:24,588 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:24,588 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:24,589 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:24,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:24,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:24,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:24,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:24,590 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:24,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:24,590 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:24,590 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:24,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:24,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:24,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:24,591 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:24,591 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c7ea2f[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:24,591 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:24,592 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:24,594 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:24,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:24,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:24,594 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:24,594 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:24,594 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:24,595 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:24,595 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:24,596 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:24,596 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:24,596 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:24,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:24,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:24,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:24,596 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:24,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:24,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:24,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:24,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:24,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:24,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:24,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:24,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:24,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:24,599 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:24,599 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:24,599 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:24,599 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:24,599 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:24,601 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:24,601 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:24,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:24,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:24,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:24,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:24,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:24,601 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:24,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:24,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:24,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:24,602 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:24,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:24,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:24,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:24,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:24,603 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:24,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:24,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:24,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:24,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:24,605 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:24,605 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:24,605 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:24,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:24,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:24,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:24,605 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:24,606 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:24,606 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:24,606 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:24,606 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:24,606 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:24,606 DEBUG [Thread-2431] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:24,606 DEBUG [Thread-2431] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:24,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:24,605 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:24,606 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:24,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:24,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:24,706 DEBUG [Thread-2431] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:24,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:24,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:24,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:24,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:24,710 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:24,710 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:24,710 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:24,710 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:24,710 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:24,710 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:24,712 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:24,712 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:24,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:24,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:24,713 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:24,713 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:24,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:24,713 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:24,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:24,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:24,714 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:24,714 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:24,714 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:24,714 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:24,714 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:24,714 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:24,716 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:24,716 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:24,716 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:24,716 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. started... 2023-04-09 09:55:24,716 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bf27154b49f0831beaa90741ce575bc9 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-09 09:55:24,716 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf27154b49f0831beaa90741ce575bc9, store=A 2023-04-09 09:55:24,717 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:24,717 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf27154b49f0831beaa90741ce575bc9, store=B 2023-04-09 09:55:24,717 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:24,717 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf27154b49f0831beaa90741ce575bc9, store=C 2023-04-09 09:55:24,717 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:24,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on bf27154b49f0831beaa90741ce575bc9 2023-04-09 09:55:24,848 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. as already flushing 2023-04-09 09:55:24,895 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:24,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32882 deadline: 1681034184891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:24,897 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:24,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32788 deadline: 1681034184892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:24,901 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:24,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32856 deadline: 1681034184894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:24,998 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:24,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32882 deadline: 1681034184996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:24,999 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:25,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32788 deadline: 1681034184998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:25,007 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:25,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32856 deadline: 1681034185002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:25,134 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230409b610a613a42f4b469b880bd03dc9f9ff to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409b610a613a42f4b469b880bd03dc9f9ff 2023-04-09 09:55:25,204 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:25,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32882 deadline: 1681034185201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:25,204 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:25,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32788 deadline: 1681034185201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:25,210 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:25,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32856 deadline: 1681034185209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:25,507 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:25,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32788 deadline: 1681034185505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:25,511 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:25,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32882 deadline: 1681034185507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:25,516 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:25,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32856 deadline: 1681034185513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:25,539 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=358, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/A/7e55f0674f8c4055bff695f1c2c03053 2023-04-09 09:55:25,959 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=358 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/B/4d47557aa244452d8b6a00a7a9854c7f 2023-04-09 09:55:26,012 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:26,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32788 deadline: 1681034186008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:26,021 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:26,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32882 deadline: 1681034186016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:26,024 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:26,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32856 deadline: 1681034186021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:26,378 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=358 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/C/95713831047a4fe0a66caf92bc158a31 2023-04-09 09:55:26,381 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/A/7e55f0674f8c4055bff695f1c2c03053 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/7e55f0674f8c4055bff695f1c2c03053 2023-04-09 09:55:26,384 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/7e55f0674f8c4055bff695f1c2c03053, entries=100, sequenceid=358, filesize=18.6 K 2023-04-09 09:55:26,385 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/B/4d47557aa244452d8b6a00a7a9854c7f as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/4d47557aa244452d8b6a00a7a9854c7f 2023-04-09 09:55:26,388 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/4d47557aa244452d8b6a00a7a9854c7f, entries=100, sequenceid=358, filesize=9.5 K 2023-04-09 09:55:26,389 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/C/95713831047a4fe0a66caf92bc158a31 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/95713831047a4fe0a66caf92bc158a31 2023-04-09 09:55:26,392 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/95713831047a4fe0a66caf92bc158a31, entries=100, sequenceid=358, filesize=9.5 K 2023-04-09 09:55:26,393 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for bf27154b49f0831beaa90741ce575bc9 in 1677ms, sequenceid=358, compaction requested=true 2023-04-09 09:55:26,393 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bf27154b49f0831beaa90741ce575bc9: 2023-04-09 09:55:26,393 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:26,393 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:26,393 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:26,393 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:26,393 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:26,393 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:26,395 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:26,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:26,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:26,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:26,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:26,396 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:26,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:26,396 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:26,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:26,396 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:26,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:26,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:26,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:26,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72cfd8c7[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:26,397 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:26,397 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:26,399 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:26,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:26,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:26,399 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:26,399 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:26,400 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:26,399 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:26,400 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:26,401 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:26,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:26,402 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:26,402 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:26,402 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:26,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:26,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:26,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:26,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:26,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:26,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:26,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:26,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:26,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:26,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:26,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:26,404 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:26,404 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:26,404 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:26,404 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:26,404 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:26,406 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:26,406 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:26,406 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:26,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:26,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:26,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:26,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:26,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:26,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:26,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:26,407 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:26,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:26,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:26,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:26,409 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:26,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:26,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:26,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:26,412 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:26,412 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:26,412 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:26,412 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:26,412 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:26,412 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:26,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:26,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:26,412 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:26,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:26,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:26,412 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:26,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:26,412 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:26,412 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:26,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:26,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:26,413 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:26,413 DEBUG [Thread-2431] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:26,413 DEBUG [Thread-2431] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:26,513 DEBUG [Thread-2431] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:26,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:26,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:26,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:26,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:26,524 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:26,524 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:26,524 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:26,524 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:26,526 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:26,526 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:26,527 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:26,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:26,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:26,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:26,528 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:26,529 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:26,529 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:26,529 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:26,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:26,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:26,529 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:26,540 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:26,540 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:26,540 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:26,540 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:26,540 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:26,546 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:26,546 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:26,547 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:26,547 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. started... 2023-04-09 09:55:26,547 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bf27154b49f0831beaa90741ce575bc9 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-09 09:55:26,547 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf27154b49f0831beaa90741ce575bc9, store=A 2023-04-09 09:55:26,547 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:26,548 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf27154b49f0831beaa90741ce575bc9, store=B 2023-04-09 09:55:26,548 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:26,548 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf27154b49f0831beaa90741ce575bc9, store=C 2023-04-09 09:55:26,548 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:26,594 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230409eafb78edf48745cf881b4edadb531579 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409eafb78edf48745cf881b4edadb531579 2023-04-09 09:55:27,015 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=377, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/A/aed07d1272364b61bc9b6c1224c1c147 2023-04-09 09:55:27,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on bf27154b49f0831beaa90741ce575bc9 2023-04-09 09:55:27,022 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. as already flushing 2023-04-09 09:55:27,024 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=377 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/B/9bbef30011294f5186c542b0312d593d 2023-04-09 09:55:27,045 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=377 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/C/bf604afbc75445c9b86d4776c849e3e9 2023-04-09 09:55:27,049 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/A/aed07d1272364b61bc9b6c1224c1c147 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/aed07d1272364b61bc9b6c1224c1c147 2023-04-09 09:55:27,053 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:27,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32882 deadline: 1681034187044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,053 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:27,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32788 deadline: 1681034187045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,054 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:27,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32856 deadline: 1681034187047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,056 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/aed07d1272364b61bc9b6c1224c1c147, entries=150, sequenceid=377, filesize=25.4 K 2023-04-09 09:55:27,056 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/B/9bbef30011294f5186c542b0312d593d as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/9bbef30011294f5186c542b0312d593d 2023-04-09 09:55:27,065 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/9bbef30011294f5186c542b0312d593d, entries=150, sequenceid=377, filesize=11.9 K 2023-04-09 09:55:27,066 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/C/bf604afbc75445c9b86d4776c849e3e9 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/bf604afbc75445c9b86d4776c849e3e9 2023-04-09 09:55:27,071 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/bf604afbc75445c9b86d4776c849e3e9, entries=150, sequenceid=377, filesize=11.9 K 2023-04-09 09:55:27,071 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for bf27154b49f0831beaa90741ce575bc9 in 524ms, sequenceid=377, compaction requested=true 2023-04-09 09:55:27,071 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bf27154b49f0831beaa90741ce575bc9: 2023-04-09 09:55:27,071 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:27,072 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:27,072 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:27,072 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,072 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:27,072 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:27,074 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:27,074 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:27,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:27,074 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:27,074 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:27,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:27,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:27,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:27,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:27,075 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:27,075 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71947322[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:27,075 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:27,076 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:27,077 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:27,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:27,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:27,077 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:27,077 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:27,077 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:27,077 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,077 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:27,079 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:27,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:27,079 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:27,080 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:27,080 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:27,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:27,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:27,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:27,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:27,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:27,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:27,085 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:27,085 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:27,085 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:27,085 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:27,085 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:27,085 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:27,090 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:27,090 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:27,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:27,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:27,090 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:27,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:27,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:27,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:27,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:27,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:27,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:27,091 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:27,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:27,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:27,092 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:27,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:27,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:27,096 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:27,096 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:27,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:27,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:27,097 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,097 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:27,097 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:27,097 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:27,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:27,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:27,097 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,097 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:27,097 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:27,097 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:27,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:27,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:27,097 DEBUG [Thread-2431] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:27,097 DEBUG [Thread-2431] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:27,097 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:27,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on bf27154b49f0831beaa90741ce575bc9 2023-04-09 09:55:27,158 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bf27154b49f0831beaa90741ce575bc9 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-09 09:55:27,158 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf27154b49f0831beaa90741ce575bc9, store=A 2023-04-09 09:55:27,158 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:27,158 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf27154b49f0831beaa90741ce575bc9, store=B 2023-04-09 09:55:27,159 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:27,159 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf27154b49f0831beaa90741ce575bc9, store=C 2023-04-09 09:55:27,159 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:27,197 DEBUG [Thread-2431] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:27,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:27,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:27,207 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:27,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32882 deadline: 1681034187203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:27,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:27,208 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:27,208 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:27,208 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:27,208 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:27,208 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304094983d4b1685d48b9b8939dcfe04224c3 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304094983d4b1685d48b9b8939dcfe04224c3 2023-04-09 09:55:27,216 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:27,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32856 deadline: 1681034187205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,216 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:27,216 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:27,216 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:27,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32788 deadline: 1681034187208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,219 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:27,219 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:27,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:27,220 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,220 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:27,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:27,221 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:27,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:27,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:27,221 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:27,225 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:27,225 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:27,225 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:27,226 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:27,226 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:27,229 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:27,229 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:27,229 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:27,229 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. started... 2023-04-09 09:55:27,229 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. as already flushing 2023-04-09 09:55:27,229 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:27,229 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:27,229 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:27,229 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,229 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:27,229 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:27,231 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,231 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:27,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:27,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:27,231 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:27,231 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:27,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:27,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:27,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:27,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,234 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=391, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/A/97461705218e4a728203ab23f705a606 2023-04-09 09:55:27,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:27,235 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:27,235 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19df8471[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:27,235 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:27,235 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:27,236 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:27,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:27,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:27,236 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:27,236 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:27,237 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:27,238 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:27,238 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:27,238 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:27,240 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,240 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:27,240 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:27,240 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:27,240 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:27,240 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:27,240 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:27,240 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:27,249 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:27,250 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:27,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:27,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:27,250 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:27,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:27,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:27,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:27,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:27,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:27,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:27,252 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:27,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:27,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:27,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:27,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:27,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:27,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,256 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,265 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,266 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:27,266 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:27,266 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,266 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:27,266 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:27,266 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:27,266 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:27,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:27,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:27,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:27,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:27,266 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:27,266 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:27,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:27,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:27,272 DEBUG [Thread-2431] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:27,272 DEBUG [Thread-2431] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:27,316 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:27,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32882 deadline: 1681034187309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,326 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:27,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32856 deadline: 1681034187318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,327 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:27,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32788 deadline: 1681034187318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,372 DEBUG [Thread-2431] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:27,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:27,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:27,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:27,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:27,380 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:27,380 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:27,380 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:27,380 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:27,381 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:27,381 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:27,382 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:27,382 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:27,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:27,382 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,382 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:27,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:27,382 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:27,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:27,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:27,383 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:27,384 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:27,384 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:27,384 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:27,384 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:27,384 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:27,392 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:27,392 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:27,392 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:27,392 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. started... 2023-04-09 09:55:27,392 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. as already flushing 2023-04-09 09:55:27,392 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:27,392 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:27,392 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:27,392 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,392 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:27,392 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:27,394 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,394 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:27,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:27,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:27,394 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:27,394 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:27,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:27,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:27,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:27,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:27,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:27,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@247483f7[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:27,395 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:27,395 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:27,398 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:27,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:27,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:27,398 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:27,398 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:27,398 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,399 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:27,398 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:27,400 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,400 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:27,400 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:27,400 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:27,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:27,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:27,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:27,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:27,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:27,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:27,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:27,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:27,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:27,405 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:27,405 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:27,405 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:27,405 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:27,405 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:27,407 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:27,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:27,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:27,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:27,407 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:27,407 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:27,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:27,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:27,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:27,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:27,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:27,409 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:27,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:27,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:27,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:27,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,410 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:27,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:27,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,412 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:27,412 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:27,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:27,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:27,412 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:27,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:27,412 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:27,413 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:27,413 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,413 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:27,413 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:27,413 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:27,412 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:27,413 DEBUG [Thread-2431] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:27,413 DEBUG [Thread-2431] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:27,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:27,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:27,513 DEBUG [Thread-2431] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:27,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:27,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:27,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:27,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:27,517 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:27,517 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:27,517 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:27,517 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:27,517 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:27,517 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:27,519 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,519 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:27,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:27,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:27,519 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,519 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:27,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:27,520 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:27,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:27,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:27,520 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:27,520 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:27,520 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:27,520 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:27,520 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:27,520 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:27,521 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:27,521 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:27,521 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:27,521 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. started... 2023-04-09 09:55:27,521 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. as already flushing 2023-04-09 09:55:27,521 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:27,521 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:27,521 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:27,521 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,521 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:27,521 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:27,522 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:27,522 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:27,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:27,522 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:27,522 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:27,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:27,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:27,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:27,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:27,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:27,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28716e5e[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:27,524 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:27,525 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:27,525 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:27,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:27,526 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:27,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:27,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32882 deadline: 1681034187518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,526 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:27,526 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:27,526 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,526 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:27,526 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:27,527 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,527 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:27,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:27,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:27,527 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:27,527 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:27,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:27,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:27,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:27,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:27,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:27,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:27,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:27,529 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:27,530 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:27,530 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:27,530 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:27,530 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:27,532 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:27,532 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:27,532 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:27,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:27,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:27,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:27,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:27,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:27,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:27,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:27,533 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:27,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:27,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:27,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:27,534 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:27,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:27,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:27,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,535 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,535 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:27,535 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:27,536 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:27,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:27,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:27,536 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:27,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:27,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:27,536 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:27,536 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:27,536 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,536 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:27,536 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:27,536 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:27,536 DEBUG [Thread-2431] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:27,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:27,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:27,536 DEBUG [Thread-2431] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:27,537 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:27,537 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:27,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32856 deadline: 1681034187528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32788 deadline: 1681034187529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,636 DEBUG [Thread-2431] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:27,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:27,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:27,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:27,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:27,640 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:27,640 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:27,640 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:27,640 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:27,640 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:27,640 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:27,641 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,641 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:27,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:27,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:27,642 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,642 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:27,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:27,642 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:27,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:27,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:27,642 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:27,643 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:27,643 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:27,643 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:27,643 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:27,643 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:27,643 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:27,643 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:27,643 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:27,643 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. started... 2023-04-09 09:55:27,643 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. as already flushing 2023-04-09 09:55:27,643 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:27,643 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:27,643 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:27,643 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,643 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:27,643 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:27,646 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,646 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:27,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:27,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:27,646 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:27,646 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:27,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:27,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:27,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:27,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:27,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:27,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7aae0511[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:27,647 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:27,648 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:27,648 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:27,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:27,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:27,649 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:27,649 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:27,649 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:27,649 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,649 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:27,650 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:27,650 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,650 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:27,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:27,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:27,650 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:27,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:27,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:27,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:27,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:27,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:27,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:27,652 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:27,652 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:27,652 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:27,652 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:27,652 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:27,652 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:27,654 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:27,654 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:27,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:27,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:27,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:27,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:27,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:27,654 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:27,654 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:27,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:27,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:27,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:27,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:27,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:27,657 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:27,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:27,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:27,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,659 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,659 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:27,659 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:27,659 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:27,660 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,660 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:27,660 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:27,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:27,660 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:27,660 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:27,660 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:27,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:27,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:27,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:27,660 DEBUG [Thread-2431] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:27,660 DEBUG [Thread-2431] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:27,660 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:27,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:27,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:27,722 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=391 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/B/8c0e97f8103b4cd68c9782c422bc335b 2023-04-09 09:55:27,732 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=391 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/C/046383ca20234eb182b5d613f999aedb 2023-04-09 09:55:27,735 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/A/97461705218e4a728203ab23f705a606 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/97461705218e4a728203ab23f705a606 2023-04-09 09:55:27,739 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/97461705218e4a728203ab23f705a606, entries=200, sequenceid=391, filesize=32.3 K 2023-04-09 09:55:27,740 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/B/8c0e97f8103b4cd68c9782c422bc335b as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/8c0e97f8103b4cd68c9782c422bc335b 2023-04-09 09:55:27,743 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/8c0e97f8103b4cd68c9782c422bc335b, entries=150, sequenceid=391, filesize=11.9 K 2023-04-09 09:55:27,744 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/C/046383ca20234eb182b5d613f999aedb as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/046383ca20234eb182b5d613f999aedb 2023-04-09 09:55:27,751 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/046383ca20234eb182b5d613f999aedb, entries=150, sequenceid=391, filesize=11.9 K 2023-04-09 09:55:27,752 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for bf27154b49f0831beaa90741ce575bc9 in 594ms, sequenceid=391, compaction requested=true 2023-04-09 09:55:27,752 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bf27154b49f0831beaa90741ce575bc9: 2023-04-09 09:55:27,752 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:55:27,752 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-09 09:55:27,752 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:55:27,752 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-09 09:55:27,752 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:55:27,754 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 130666 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-09 09:55:27,754 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1912): bf27154b49f0831beaa90741ce575bc9/A is initiating minor compaction (all files) 2023-04-09 09:55:27,754 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bf27154b49f0831beaa90741ce575bc9/A in TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:27,754 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/7512c91429784267933243c027150373, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/c25885a6d5d14e2d894d1c6a7738bec8, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/7e55f0674f8c4055bff695f1c2c03053, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/aed07d1272364b61bc9b6c1224c1c147, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/97461705218e4a728203ab23f705a606] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp, totalSize=127.6 K 2023-04-09 09:55:27,754 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 58930 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-09 09:55:27,754 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): bf27154b49f0831beaa90741ce575bc9/B is initiating minor compaction (all files) 2023-04-09 09:55:27,754 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bf27154b49f0831beaa90741ce575bc9/B in TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:27,754 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/f0d4f99efe0449e58ab816307ca3b37c, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/75b37fef0ce8492180570b1a1cdd336d, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/4d47557aa244452d8b6a00a7a9854c7f, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/9bbef30011294f5186c542b0312d593d, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/8c0e97f8103b4cd68c9782c422bc335b] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp, totalSize=57.5 K 2023-04-09 09:55:27,754 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting f0d4f99efe0449e58ab816307ca3b37c, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=321, earliestPutTs=1681034121179 2023-04-09 09:55:27,755 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 7512c91429784267933243c027150373, keycount=150, bloomtype=ROW, size=25.9 K, encoding=NONE, compression=NONE, seqNum=321, earliestPutTs=1681034121179 2023-04-09 09:55:27,755 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 75b37fef0ce8492180570b1a1cdd336d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=343, earliestPutTs=1681034121554 2023-04-09 09:55:27,755 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting c25885a6d5d14e2d894d1c6a7738bec8, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=343, earliestPutTs=1681034121554 2023-04-09 09:55:27,756 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 7e55f0674f8c4055bff695f1c2c03053, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=358, earliestPutTs=1681034123703 2023-04-09 09:55:27,756 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 4d47557aa244452d8b6a00a7a9854c7f, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=358, earliestPutTs=1681034123703 2023-04-09 09:55:27,756 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting aed07d1272364b61bc9b6c1224c1c147, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=377, earliestPutTs=1681034124885 2023-04-09 09:55:27,756 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 9bbef30011294f5186c542b0312d593d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=377, earliestPutTs=1681034124885 2023-04-09 09:55:27,757 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 97461705218e4a728203ab23f705a606, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=391, earliestPutTs=1681034127043 2023-04-09 09:55:27,757 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 8c0e97f8103b4cd68c9782c422bc335b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=391, earliestPutTs=1681034127043 2023-04-09 09:55:27,760 DEBUG [Thread-2431] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:27,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:27,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:27,770 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): bf27154b49f0831beaa90741ce575bc9#B#compaction#382 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:55:27,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:27,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:27,772 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:27,772 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:27,772 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:27,772 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:27,773 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:27,773 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:27,774 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,774 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:27,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:27,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:27,774 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:27,774 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:27,775 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:27,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:27,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:27,775 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:27,788 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:27,788 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:27,788 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:27,788 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:27,788 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:27,791 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] throttle.PressureAwareThroughputController(145): bf27154b49f0831beaa90741ce575bc9#A#compaction#381 average throughput is 0.79 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:55:27,792 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:27,792 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:27,792 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:27,792 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. started... 2023-04-09 09:55:27,792 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bf27154b49f0831beaa90741ce575bc9 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-09 09:55:27,792 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf27154b49f0831beaa90741ce575bc9, store=A 2023-04-09 09:55:27,793 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:27,793 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf27154b49f0831beaa90741ce575bc9, store=B 2023-04-09 09:55:27,793 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:27,793 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf27154b49f0831beaa90741ce575bc9, store=C 2023-04-09 09:55:27,793 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:27,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on bf27154b49f0831beaa90741ce575bc9 2023-04-09 09:55:27,850 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. as already flushing 2023-04-09 09:55:27,902 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:27,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32882 deadline: 1681034187890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,903 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:27,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32856 deadline: 1681034187891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:27,904 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:27,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32788 deadline: 1681034187892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:28,009 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:28,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32882 deadline: 1681034188004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:28,009 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:28,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32856 deadline: 1681034188004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:28,009 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:28,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32788 deadline: 1681034188005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:28,210 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/B/129d769e7e83484391fac9f78a9c145a as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/129d769e7e83484391fac9f78a9c145a 2023-04-09 09:55:28,214 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in bf27154b49f0831beaa90741ce575bc9/B of bf27154b49f0831beaa90741ce575bc9 into 129d769e7e83484391fac9f78a9c145a(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:55:28,214 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bf27154b49f0831beaa90741ce575bc9: 2023-04-09 09:55:28,214 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9., storeName=bf27154b49f0831beaa90741ce575bc9/B, priority=11, startTime=1681034127752; duration=0sec 2023-04-09 09:55:28,214 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:55:28,214 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-09 09:55:28,215 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 58930 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-09 09:55:28,215 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): bf27154b49f0831beaa90741ce575bc9/C is initiating minor compaction (all files) 2023-04-09 09:55:28,215 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bf27154b49f0831beaa90741ce575bc9/C in TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:28,215 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/b9cff78f8f5f42a08da671626df1c61e, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/9bc340e4dd5c4eb7a24905b6dd04a105, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/95713831047a4fe0a66caf92bc158a31, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/bf604afbc75445c9b86d4776c849e3e9, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/046383ca20234eb182b5d613f999aedb] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp, totalSize=57.5 K 2023-04-09 09:55:28,216 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting b9cff78f8f5f42a08da671626df1c61e, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=321, earliestPutTs=1681034121179 2023-04-09 09:55:28,216 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:28,216 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 9bc340e4dd5c4eb7a24905b6dd04a105, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=343, earliestPutTs=1681034121554 2023-04-09 09:55:28,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32882 deadline: 1681034188210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:28,216 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:28,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32856 deadline: 1681034188210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:28,217 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 95713831047a4fe0a66caf92bc158a31, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=358, earliestPutTs=1681034123703 2023-04-09 09:55:28,217 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting bf604afbc75445c9b86d4776c849e3e9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=377, earliestPutTs=1681034124885 2023-04-09 09:55:28,217 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 046383ca20234eb182b5d613f999aedb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=391, earliestPutTs=1681034127043 2023-04-09 09:55:28,220 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:28,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32788 deadline: 1681034188211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:28,226 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): bf27154b49f0831beaa90741ce575bc9#C#compaction#384 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:55:28,274 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304092f226f3a05aa45729f0b6bc5a69326f0 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304092f226f3a05aa45729f0b6bc5a69326f0 2023-04-09 09:55:28,279 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=409, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/A/f1f0c8d6c2f54f0fbc8c6222c5ce3945 2023-04-09 09:55:28,522 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:28,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32856 deadline: 1681034188518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:28,522 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:28,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32882 deadline: 1681034188519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:28,527 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:28,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32788 deadline: 1681034188523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:28,636 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/C/602443387b5b4e1abab3773aff5b4b9a as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/602443387b5b4e1abab3773aff5b4b9a 2023-04-09 09:55:28,640 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in bf27154b49f0831beaa90741ce575bc9/C of bf27154b49f0831beaa90741ce575bc9 into 602443387b5b4e1abab3773aff5b4b9a(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:55:28,640 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bf27154b49f0831beaa90741ce575bc9: 2023-04-09 09:55:28,640 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9., storeName=bf27154b49f0831beaa90741ce575bc9/C, priority=11, startTime=1681034127752; duration=0sec 2023-04-09 09:55:28,640 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:55:28,655 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/A/980054979d6247109a79cff71f6cc19e as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/980054979d6247109a79cff71f6cc19e 2023-04-09 09:55:28,659 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in bf27154b49f0831beaa90741ce575bc9/A of bf27154b49f0831beaa90741ce575bc9 into 980054979d6247109a79cff71f6cc19e(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:55:28,659 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bf27154b49f0831beaa90741ce575bc9: 2023-04-09 09:55:28,659 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9., storeName=bf27154b49f0831beaa90741ce575bc9/A, priority=11, startTime=1681034127752; duration=0sec 2023-04-09 09:55:28,659 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:55:28,693 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=409 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/B/e9b11377a989482a9f357714bb4854f1 2023-04-09 09:55:28,702 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=409 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/C/a89644051cc447d8bf9c0661d7eeedb2 2023-04-09 09:55:28,705 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/A/f1f0c8d6c2f54f0fbc8c6222c5ce3945 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/f1f0c8d6c2f54f0fbc8c6222c5ce3945 2023-04-09 09:55:28,709 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/f1f0c8d6c2f54f0fbc8c6222c5ce3945, entries=150, sequenceid=409, filesize=25.4 K 2023-04-09 09:55:28,709 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/B/e9b11377a989482a9f357714bb4854f1 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/e9b11377a989482a9f357714bb4854f1 2023-04-09 09:55:28,714 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/e9b11377a989482a9f357714bb4854f1, entries=150, sequenceid=409, filesize=11.9 K 2023-04-09 09:55:28,714 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/C/a89644051cc447d8bf9c0661d7eeedb2 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/a89644051cc447d8bf9c0661d7eeedb2 2023-04-09 09:55:28,717 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/a89644051cc447d8bf9c0661d7eeedb2, entries=150, sequenceid=409, filesize=11.9 K 2023-04-09 09:55:28,718 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for bf27154b49f0831beaa90741ce575bc9 in 926ms, sequenceid=409, compaction requested=false 2023-04-09 09:55:28,718 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bf27154b49f0831beaa90741ce575bc9: 2023-04-09 09:55:28,718 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:28,718 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:28,718 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:28,718 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:28,718 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:28,718 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:28,721 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:28,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:28,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:28,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:28,721 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:28,721 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:28,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:28,721 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:28,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:28,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:28,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:28,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:28,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:28,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1fb7ea39[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:28,722 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:28,723 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:28,723 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:28,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:28,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:28,724 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:28,724 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:28,724 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:28,724 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:28,724 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:28,725 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:28,725 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:28,725 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:28,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:28,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:28,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:28,725 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:28,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:28,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:28,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:28,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:28,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:28,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:28,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:28,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:28,728 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:28,728 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:28,728 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:28,728 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:28,728 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:28,728 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:28,730 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:28,730 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:28,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:28,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:28,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:28,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:28,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:28,730 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:28,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:28,730 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:28,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:28,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:28,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:28,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:28,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:28,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:28,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:28,732 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:28,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:28,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:28,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:28,734 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:28,734 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:28,734 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:28,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:28,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:28,735 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:28,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:28,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:28,735 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:28,735 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:28,735 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:28,735 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:28,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:28,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:28,735 DEBUG [Thread-2431] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:28,735 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:28,735 DEBUG [Thread-2431] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:28,735 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:28,735 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:28,835 DEBUG [Thread-2431] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:28,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:28,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:28,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:28,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:28,840 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:28,840 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:28,840 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:28,840 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:28,840 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:28,840 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:28,842 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:28,842 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:28,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:28,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:28,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:28,842 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:28,842 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:28,843 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:28,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:28,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:28,843 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:28,844 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:28,844 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:28,844 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:28,844 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:28,844 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:28,844 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:28,844 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:28,844 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:28,845 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. started... 2023-04-09 09:55:28,845 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bf27154b49f0831beaa90741ce575bc9 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-09 09:55:28,845 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf27154b49f0831beaa90741ce575bc9, store=A 2023-04-09 09:55:28,845 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:28,845 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf27154b49f0831beaa90741ce575bc9, store=B 2023-04-09 09:55:28,845 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:28,845 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf27154b49f0831beaa90741ce575bc9, store=C 2023-04-09 09:55:28,845 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:28,867 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230409537ea8f5b5c04020a4a6ad1a8739be0b to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409537ea8f5b5c04020a4a6ad1a8739be0b 2023-04-09 09:55:29,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on bf27154b49f0831beaa90741ce575bc9 2023-04-09 09:55:29,025 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. as already flushing 2023-04-09 09:55:29,050 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:29,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32788 deadline: 1681034189045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:29,050 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:29,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32856 deadline: 1681034189047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:29,054 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:29,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32882 deadline: 1681034189049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:29,154 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:29,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32788 deadline: 1681034189151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:29,155 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:29,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32856 deadline: 1681034189151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:29,160 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:29,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32882 deadline: 1681034189155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:29,274 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=427, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/A/3d7337c77ae7439391a3cf5cc90280f4 2023-04-09 09:55:29,359 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:29,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32788 deadline: 1681034189356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:29,359 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:29,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32856 deadline: 1681034189357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:29,365 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:29,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32882 deadline: 1681034189361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:29,664 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:29,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32788 deadline: 1681034189662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:29,665 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:29,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32856 deadline: 1681034189662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:29,670 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:29,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32882 deadline: 1681034189666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:29,683 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=427 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/B/b33dc6b1b4a845d5b560a653ca07cb84 2023-04-09 09:55:30,110 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=427 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/C/80120736f84a456681e912b570cb30b0 2023-04-09 09:55:30,114 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/A/3d7337c77ae7439391a3cf5cc90280f4 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/3d7337c77ae7439391a3cf5cc90280f4 2023-04-09 09:55:30,116 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/3d7337c77ae7439391a3cf5cc90280f4, entries=150, sequenceid=427, filesize=25.4 K 2023-04-09 09:55:30,117 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/B/b33dc6b1b4a845d5b560a653ca07cb84 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/b33dc6b1b4a845d5b560a653ca07cb84 2023-04-09 09:55:30,120 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/b33dc6b1b4a845d5b560a653ca07cb84, entries=150, sequenceid=427, filesize=11.9 K 2023-04-09 09:55:30,121 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/C/80120736f84a456681e912b570cb30b0 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/80120736f84a456681e912b570cb30b0 2023-04-09 09:55:30,124 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/80120736f84a456681e912b570cb30b0, entries=150, sequenceid=427, filesize=11.9 K 2023-04-09 09:55:30,125 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for bf27154b49f0831beaa90741ce575bc9 in 1280ms, sequenceid=427, compaction requested=true 2023-04-09 09:55:30,125 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bf27154b49f0831beaa90741ce575bc9: 2023-04-09 09:55:30,125 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:30,125 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:30,125 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:30,125 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:30,125 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:30,125 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:30,131 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:30,131 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:30,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:30,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:30,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:30,131 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:30,131 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:30,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:30,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:30,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:30,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:30,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:30,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:30,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f5cb903[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:30,133 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:30,133 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:30,134 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:30,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:30,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:30,134 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:30,134 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:30,134 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:30,134 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:30,134 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:30,135 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:30,135 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:30,135 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:30,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:30,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:30,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:30,135 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:30,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:30,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:30,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:30,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:30,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:30,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:30,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:30,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:30,138 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:30,138 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:30,138 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:30,138 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:30,138 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:30,138 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:30,141 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:30,141 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:30,141 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:30,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:30,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:30,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:30,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:30,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:30,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:30,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:30,141 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:30,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:30,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:30,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:30,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:30,143 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:30,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:30,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:30,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:30,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:30,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:30,146 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:30,146 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:30,148 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:30,148 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:30,148 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:30,148 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:30,148 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:30,148 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:30,148 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:30,148 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:30,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:30,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:30,148 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:30,149 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:30,148 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:30,149 DEBUG [Thread-2431] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:30,149 DEBUG [Thread-2431] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:30,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:30,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:30,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on bf27154b49f0831beaa90741ce575bc9 2023-04-09 09:55:30,173 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bf27154b49f0831beaa90741ce575bc9 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-09 09:55:30,173 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf27154b49f0831beaa90741ce575bc9, store=A 2023-04-09 09:55:30,173 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:30,173 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf27154b49f0831beaa90741ce575bc9, store=B 2023-04-09 09:55:30,173 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:30,174 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf27154b49f0831beaa90741ce575bc9, store=C 2023-04-09 09:55:30,174 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:30,185 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230409bfb87e24b4384681ab42a7df1062659a to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409bfb87e24b4384681ab42a7df1062659a 2023-04-09 09:55:30,190 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=446, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/A/588ab4fdf564493e9acbaeb51b95896c 2023-04-09 09:55:30,196 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:30,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32788 deadline: 1681034190190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:30,198 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:30,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32856 deadline: 1681034190191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:30,198 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:30,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32882 deadline: 1681034190195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:30,200 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=446 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/B/7ef6f5ddbf5f43fdb9650f7e70170dcf 2023-04-09 09:55:30,209 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=446 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/C/75ca0c8dcac1435ea55245628925338e 2023-04-09 09:55:30,214 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/A/588ab4fdf564493e9acbaeb51b95896c as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/588ab4fdf564493e9acbaeb51b95896c 2023-04-09 09:55:30,217 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/588ab4fdf564493e9acbaeb51b95896c, entries=250, sequenceid=446, filesize=39.2 K 2023-04-09 09:55:30,218 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/B/7ef6f5ddbf5f43fdb9650f7e70170dcf as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/7ef6f5ddbf5f43fdb9650f7e70170dcf 2023-04-09 09:55:30,220 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/7ef6f5ddbf5f43fdb9650f7e70170dcf, entries=150, sequenceid=446, filesize=11.9 K 2023-04-09 09:55:30,221 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/C/75ca0c8dcac1435ea55245628925338e as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/75ca0c8dcac1435ea55245628925338e 2023-04-09 09:55:30,225 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/75ca0c8dcac1435ea55245628925338e, entries=150, sequenceid=446, filesize=11.9 K 2023-04-09 09:55:30,226 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for bf27154b49f0831beaa90741ce575bc9 in 53ms, sequenceid=446, compaction requested=true 2023-04-09 09:55:30,226 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bf27154b49f0831beaa90741ce575bc9: 2023-04-09 09:55:30,226 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:55:30,226 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-09 09:55:30,226 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-09 09:55:30,226 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:55:30,226 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:55:30,227 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 118882 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-09 09:55:30,227 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1912): bf27154b49f0831beaa90741ce575bc9/A is initiating minor compaction (all files) 2023-04-09 09:55:30,227 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bf27154b49f0831beaa90741ce575bc9/A in TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:30,227 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49358 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-09 09:55:30,227 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/980054979d6247109a79cff71f6cc19e, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/f1f0c8d6c2f54f0fbc8c6222c5ce3945, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/3d7337c77ae7439391a3cf5cc90280f4, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/588ab4fdf564493e9acbaeb51b95896c] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp, totalSize=116.1 K 2023-04-09 09:55:30,227 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): bf27154b49f0831beaa90741ce575bc9/B is initiating minor compaction (all files) 2023-04-09 09:55:30,227 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bf27154b49f0831beaa90741ce575bc9/B in TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:30,227 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/129d769e7e83484391fac9f78a9c145a, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/e9b11377a989482a9f357714bb4854f1, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/b33dc6b1b4a845d5b560a653ca07cb84, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/7ef6f5ddbf5f43fdb9650f7e70170dcf] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp, totalSize=48.2 K 2023-04-09 09:55:30,227 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 980054979d6247109a79cff71f6cc19e, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=391, earliestPutTs=1681034127043 2023-04-09 09:55:30,228 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 129d769e7e83484391fac9f78a9c145a, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=391, earliestPutTs=1681034127043 2023-04-09 09:55:30,228 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting f1f0c8d6c2f54f0fbc8c6222c5ce3945, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=409, earliestPutTs=1681034127192 2023-04-09 09:55:30,228 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting e9b11377a989482a9f357714bb4854f1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=409, earliestPutTs=1681034127192 2023-04-09 09:55:30,228 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 3d7337c77ae7439391a3cf5cc90280f4, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=427, earliestPutTs=1681034127876 2023-04-09 09:55:30,228 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting b33dc6b1b4a845d5b560a653ca07cb84, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=427, earliestPutTs=1681034127876 2023-04-09 09:55:30,228 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 588ab4fdf564493e9acbaeb51b95896c, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=446, earliestPutTs=1681034129036 2023-04-09 09:55:30,228 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 7ef6f5ddbf5f43fdb9650f7e70170dcf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=446, earliestPutTs=1681034129043 2023-04-09 09:55:30,237 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): bf27154b49f0831beaa90741ce575bc9#B#compaction#394 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:55:30,240 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] throttle.PressureAwareThroughputController(145): bf27154b49f0831beaa90741ce575bc9#A#compaction#393 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:55:30,249 DEBUG [Thread-2431] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:30,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:30,251 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:30,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:30,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:30,254 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:30,254 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:30,254 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:30,254 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:30,254 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:30,254 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:30,258 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:30,258 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:30,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:30,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:30,258 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:30,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:30,258 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:30,258 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:30,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:30,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:30,259 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:30,259 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:30,259 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:30,259 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:30,259 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:30,259 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:30,260 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:30,260 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:30,260 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:30,260 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. started... 2023-04-09 09:55:30,260 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bf27154b49f0831beaa90741ce575bc9 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-09 09:55:30,260 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf27154b49f0831beaa90741ce575bc9, store=A 2023-04-09 09:55:30,260 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:30,260 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf27154b49f0831beaa90741ce575bc9, store=B 2023-04-09 09:55:30,260 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:30,260 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf27154b49f0831beaa90741ce575bc9, store=C 2023-04-09 09:55:30,260 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:30,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on bf27154b49f0831beaa90741ce575bc9 2023-04-09 09:55:30,301 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. as already flushing 2023-04-09 09:55:30,347 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:30,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32882 deadline: 1681034190338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:30,347 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:30,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32856 deadline: 1681034190339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:30,349 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:30,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32788 deadline: 1681034190340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:30,450 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:30,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32882 deadline: 1681034190448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:30,451 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:30,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32856 deadline: 1681034190448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:30,455 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:30,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32788 deadline: 1681034190450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:30,645 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/B/39dae982f0034f888603ec8aad6c4249 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/39dae982f0034f888603ec8aad6c4249 2023-04-09 09:55:30,649 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in bf27154b49f0831beaa90741ce575bc9/B of bf27154b49f0831beaa90741ce575bc9 into 39dae982f0034f888603ec8aad6c4249(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:55:30,649 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bf27154b49f0831beaa90741ce575bc9: 2023-04-09 09:55:30,649 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9., storeName=bf27154b49f0831beaa90741ce575bc9/B, priority=12, startTime=1681034130226; duration=0sec 2023-04-09 09:55:30,649 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:55:30,649 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-09 09:55:30,650 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49358 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-09 09:55:30,650 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): bf27154b49f0831beaa90741ce575bc9/C is initiating minor compaction (all files) 2023-04-09 09:55:30,650 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bf27154b49f0831beaa90741ce575bc9/C in TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:30,650 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/602443387b5b4e1abab3773aff5b4b9a, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/a89644051cc447d8bf9c0661d7eeedb2, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/80120736f84a456681e912b570cb30b0, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/75ca0c8dcac1435ea55245628925338e] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp, totalSize=48.2 K 2023-04-09 09:55:30,651 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 602443387b5b4e1abab3773aff5b4b9a, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=391, earliestPutTs=1681034127043 2023-04-09 09:55:30,651 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting a89644051cc447d8bf9c0661d7eeedb2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=409, earliestPutTs=1681034127192 2023-04-09 09:55:30,651 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 80120736f84a456681e912b570cb30b0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=427, earliestPutTs=1681034127876 2023-04-09 09:55:30,652 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 75ca0c8dcac1435ea55245628925338e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=446, earliestPutTs=1681034129043 2023-04-09 09:55:30,659 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:30,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32882 deadline: 1681034190651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:30,659 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): bf27154b49f0831beaa90741ce575bc9#C#compaction#396 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:55:30,659 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:30,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32856 deadline: 1681034190651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:30,663 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:30,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32788 deadline: 1681034190656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:30,667 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/A/eaf3a41bda4b44748367956245961c2a as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/eaf3a41bda4b44748367956245961c2a 2023-04-09 09:55:30,673 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in bf27154b49f0831beaa90741ce575bc9/A of bf27154b49f0831beaa90741ce575bc9 into eaf3a41bda4b44748367956245961c2a(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:55:30,673 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bf27154b49f0831beaa90741ce575bc9: 2023-04-09 09:55:30,673 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/C/1b96b68a8c064b63aecee8c4fd917123 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/1b96b68a8c064b63aecee8c4fd917123 2023-04-09 09:55:30,673 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9., storeName=bf27154b49f0831beaa90741ce575bc9/A, priority=12, startTime=1681034130226; duration=0sec 2023-04-09 09:55:30,673 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:55:30,678 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in bf27154b49f0831beaa90741ce575bc9/C of bf27154b49f0831beaa90741ce575bc9 into 1b96b68a8c064b63aecee8c4fd917123(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:55:30,678 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bf27154b49f0831beaa90741ce575bc9: 2023-04-09 09:55:30,678 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9., storeName=bf27154b49f0831beaa90741ce575bc9/C, priority=12, startTime=1681034130226; duration=0sec 2023-04-09 09:55:30,678 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:55:30,680 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304090ba359b4f7bb40bc939477e32f5f37e3 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304090ba359b4f7bb40bc939477e32f5f37e3 2023-04-09 09:55:30,960 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:30,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32882 deadline: 1681034190960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:30,964 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:30,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32856 deadline: 1681034190961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:30,969 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:30,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32788 deadline: 1681034190966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:31,089 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=459, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/A/e1f848d9dd494b3c9466f10c56b80d2e 2023-04-09 09:55:31,468 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:31,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32882 deadline: 1681034191465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:31,474 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:31,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32856 deadline: 1681034191469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:31,477 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:31,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32788 deadline: 1681034191473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:31,499 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=459 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/B/f6e68e8daa704354a4331786e188bdf0 2023-04-09 09:55:31,509 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=459 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/C/ca84d2214aea419e9a75df16b769dc83 2023-04-09 09:55:31,513 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/A/e1f848d9dd494b3c9466f10c56b80d2e as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/e1f848d9dd494b3c9466f10c56b80d2e 2023-04-09 09:55:31,516 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/e1f848d9dd494b3c9466f10c56b80d2e, entries=150, sequenceid=459, filesize=25.4 K 2023-04-09 09:55:31,517 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/B/f6e68e8daa704354a4331786e188bdf0 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/f6e68e8daa704354a4331786e188bdf0 2023-04-09 09:55:31,520 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/f6e68e8daa704354a4331786e188bdf0, entries=150, sequenceid=459, filesize=11.9 K 2023-04-09 09:55:31,521 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/C/ca84d2214aea419e9a75df16b769dc83 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/ca84d2214aea419e9a75df16b769dc83 2023-04-09 09:55:31,524 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/ca84d2214aea419e9a75df16b769dc83, entries=150, sequenceid=459, filesize=11.9 K 2023-04-09 09:55:31,525 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for bf27154b49f0831beaa90741ce575bc9 in 1265ms, sequenceid=459, compaction requested=false 2023-04-09 09:55:31,525 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bf27154b49f0831beaa90741ce575bc9: 2023-04-09 09:55:31,525 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:31,525 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:31,525 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:31,525 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:31,525 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:31,525 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:31,527 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:31,527 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:31,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:31,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:31,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:31,527 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:31,527 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:31,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:31,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:31,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:31,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:31,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:31,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:31,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cfc3728[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:31,529 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:31,529 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:31,530 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:31,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:31,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:31,530 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:31,530 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:31,530 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:31,530 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:31,530 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:31,531 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:31,531 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:31,531 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:31,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:31,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:31,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:31,531 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:31,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:31,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:31,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:31,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:31,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:31,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:31,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:31,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:31,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:31,533 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:31,533 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:31,533 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:31,533 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:31,533 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:31,535 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:31,535 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:31,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:31,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:31,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:31,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:31,535 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:31,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:31,535 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:31,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:31,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:31,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:31,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:31,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:31,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:31,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:31,537 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:31,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:31,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:31,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:31,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:31,540 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:31,540 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:31,540 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:31,540 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:31,540 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:31,540 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:31,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:31,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:31,540 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:31,540 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:31,541 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:31,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:31,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:31,541 DEBUG [Thread-2431] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:31,541 DEBUG [Thread-2431] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:31,541 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:31,541 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:31,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:31,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:31,641 DEBUG [Thread-2431] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:31,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:31,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:31,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:31,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:31,646 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:31,646 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:31,646 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:31,646 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:31,646 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:31,646 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:31,647 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:31,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:31,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:31,648 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:31,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:31,648 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:31,648 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:31,648 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:31,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:31,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:31,649 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:31,649 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:31,649 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:31,649 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:31,649 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:31,649 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:31,649 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:31,649 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:31,649 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:31,649 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. started... 2023-04-09 09:55:31,649 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bf27154b49f0831beaa90741ce575bc9 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-09 09:55:31,649 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf27154b49f0831beaa90741ce575bc9, store=A 2023-04-09 09:55:31,649 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:31,650 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf27154b49f0831beaa90741ce575bc9, store=B 2023-04-09 09:55:31,650 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:31,650 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf27154b49f0831beaa90741ce575bc9, store=C 2023-04-09 09:55:31,650 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:31,664 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230409a5ecae688e414b5780cc878771a83703 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409a5ecae688e414b5780cc878771a83703 2023-04-09 09:55:32,069 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=481, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/A/a1a77ec971704dcaae9592c69ec90eb9 2023-04-09 09:55:32,079 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=481 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/B/a4542f1050774377973fb7c67c2092b6 2023-04-09 09:55:32,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on bf27154b49f0831beaa90741ce575bc9 2023-04-09 09:55:32,473 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. as already flushing 2023-04-09 09:55:32,489 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=481 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/C/774b1a0a496b4f3c8c2a310478cc074a 2023-04-09 09:55:32,492 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/A/a1a77ec971704dcaae9592c69ec90eb9 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/a1a77ec971704dcaae9592c69ec90eb9 2023-04-09 09:55:32,495 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/a1a77ec971704dcaae9592c69ec90eb9, entries=150, sequenceid=481, filesize=25.4 K 2023-04-09 09:55:32,496 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/B/a4542f1050774377973fb7c67c2092b6 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/a4542f1050774377973fb7c67c2092b6 2023-04-09 09:55:32,498 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:32,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32856 deadline: 1681034192490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,499 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:32,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32882 deadline: 1681034192491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,500 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/a4542f1050774377973fb7c67c2092b6, entries=150, sequenceid=481, filesize=11.9 K 2023-04-09 09:55:32,500 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/C/774b1a0a496b4f3c8c2a310478cc074a as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/774b1a0a496b4f3c8c2a310478cc074a 2023-04-09 09:55:32,503 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/774b1a0a496b4f3c8c2a310478cc074a, entries=150, sequenceid=481, filesize=11.9 K 2023-04-09 09:55:32,504 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for bf27154b49f0831beaa90741ce575bc9 in 855ms, sequenceid=481, compaction requested=true 2023-04-09 09:55:32,504 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bf27154b49f0831beaa90741ce575bc9: 2023-04-09 09:55:32,504 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:32,504 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:32,504 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:32,504 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,504 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:32,504 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:32,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on bf27154b49f0831beaa90741ce575bc9 2023-04-09 09:55:32,505 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bf27154b49f0831beaa90741ce575bc9 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-09 09:55:32,505 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf27154b49f0831beaa90741ce575bc9, store=A 2023-04-09 09:55:32,506 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:32,506 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf27154b49f0831beaa90741ce575bc9, store=B 2023-04-09 09:55:32,506 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:32,506 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf27154b49f0831beaa90741ce575bc9, store=C 2023-04-09 09:55:32,506 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,506 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:32,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:32,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:32,506 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:32,506 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:32,506 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:32,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:32,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:32,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:32,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:32,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:32,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36f03b2b[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:32,508 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:32,508 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:32,509 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:32,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:32,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:32,509 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:32,509 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:32,509 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:32,510 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,510 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:32,511 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:32,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:32,511 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:32,511 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:32,511 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:32,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:32,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:32,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:32,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:32,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:32,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:32,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:32,514 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:32,515 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:32,515 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:32,515 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:32,515 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:32,517 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304094133548ab9af49d68183cf2b4a606b4c to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304094133548ab9af49d68183cf2b4a606b4c 2023-04-09 09:55:32,518 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:32,518 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:32,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:32,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:32,518 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:32,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:32,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:32,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:32,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:32,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:32,518 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:32,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:32,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:32,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:32,520 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:32,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:32,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:32,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,522 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,522 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:32,522 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:32,522 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,522 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:32,522 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:32,522 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:32,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:32,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:32,522 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:32,522 DEBUG [Thread-2431] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:32,522 DEBUG [Thread-2431] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:32,522 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:32,524 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:32,524 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:32,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:32,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:32,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:32,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:32,565 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:32,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32870 deadline: 1681034192560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,567 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:32,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32772 deadline: 1681034192561, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,567 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:32,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32788 deadline: 1681034192562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,606 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:32,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32856 deadline: 1681034192599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,606 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:32,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32882 deadline: 1681034192600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,622 DEBUG [Thread-2431] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:32,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:32,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:32,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:32,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:32,626 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:32,626 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:32,626 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:32,626 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:32,626 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:32,626 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:32,627 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,627 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:32,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:32,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:32,628 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,628 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:32,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:32,628 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:32,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:32,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:32,629 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:32,629 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:32,629 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:32,629 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:32,629 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:32,629 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:32,629 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:32,629 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:32,629 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:32,629 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. started... 2023-04-09 09:55:32,629 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. as already flushing 2023-04-09 09:55:32,629 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:32,629 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:32,629 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:32,629 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,629 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:32,630 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:32,631 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,631 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:32,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:32,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:32,631 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:32,631 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:32,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:32,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:32,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:32,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:32,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:32,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16b6e9ee[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:32,633 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:32,633 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:32,635 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:32,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:32,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:32,635 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:32,635 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:32,635 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,635 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:32,635 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:32,636 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,636 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:32,636 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:32,637 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:32,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:32,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:32,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:32,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:32,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:32,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:32,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:32,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:32,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:32,639 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:32,639 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:32,639 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:32,639 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:32,639 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:32,640 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:32,640 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:32,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:32,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:32,640 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:32,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:32,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:32,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:32,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:32,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:32,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:32,641 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:32,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:32,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:32,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:32,642 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:32,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:32,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,645 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,645 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:32,645 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:32,645 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:32,645 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:32,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:32,645 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:32,645 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:32,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:32,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:32,645 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:32,645 DEBUG [Thread-2431] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:32,645 DEBUG [Thread-2431] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:32,645 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:32,645 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:32,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:32,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:32,670 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:32,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32870 deadline: 1681034192666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,671 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:32,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32772 deadline: 1681034192668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,672 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:32,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32788 deadline: 1681034192668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,745 DEBUG [Thread-2431] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:32,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:32,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:32,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:32,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:32,749 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:32,749 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:32,749 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:32,749 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:32,750 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:32,750 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:32,753 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,753 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:32,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:32,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:32,753 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,753 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:32,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:32,753 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:32,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:32,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:32,754 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:32,754 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:32,754 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:32,754 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:32,754 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:32,754 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:32,754 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:32,754 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:32,754 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:32,754 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. started... 2023-04-09 09:55:32,755 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. as already flushing 2023-04-09 09:55:32,755 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:32,755 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:32,755 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:32,755 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,755 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:32,755 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:32,759 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,759 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:32,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:32,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:32,759 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:32,759 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:32,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:32,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:32,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:32,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:32,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:32,760 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:32,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c58de9a[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:32,760 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:32,761 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:32,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:32,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:32,761 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:32,761 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,762 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:32,762 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:32,762 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:32,764 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:32,764 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,764 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:32,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:32,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:32,764 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:32,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:32,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:32,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:32,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:32,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:32,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:32,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:32,766 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:32,767 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:32,767 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:32,767 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:32,767 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:32,768 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:32,768 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:32,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:32,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:32,768 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:32,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:32,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:32,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:32,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:32,768 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:32,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:32,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:32,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:32,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:32,771 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:32,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:32,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:32,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,774 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,774 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:32,774 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:32,774 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,774 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:32,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:32,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:32,774 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:32,775 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:32,775 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:32,775 DEBUG [Thread-2431] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:32,775 DEBUG [Thread-2431] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:32,775 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:32,780 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:32,780 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:32,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:32,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:32,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:32,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:32,809 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:32,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32882 deadline: 1681034192807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,810 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:32,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32856 deadline: 1681034192808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,875 DEBUG [Thread-2431] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:32,875 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:32,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32870 deadline: 1681034192871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:32,876 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:32,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32772 deadline: 1681034192872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,876 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:32,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32788 deadline: 1681034192872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,877 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:32,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:32,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:32,878 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:32,879 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:32,879 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:32,879 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:32,879 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:32,879 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:32,882 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:32,882 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:32,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:32,882 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,882 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:32,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:32,883 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:32,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:32,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:32,883 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:32,883 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:32,884 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:32,884 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:32,884 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:32,884 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:32,884 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:32,884 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:32,884 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:32,884 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. started... 2023-04-09 09:55:32,884 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. as already flushing 2023-04-09 09:55:32,884 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:32,884 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:32,884 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:32,884 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,884 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:32,884 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:32,886 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,886 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:32,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:32,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:32,887 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:32,887 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:32,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:32,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:32,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:32,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:32,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:32,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31c0e14e[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:32,888 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:32,888 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:32,889 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:32,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:32,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:32,889 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:32,889 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:32,889 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,889 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:32,889 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:32,891 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,891 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:32,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:32,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:32,891 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:32,891 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:32,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:32,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:32,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:32,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:32,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:32,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:32,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:32,893 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:32,893 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:32,893 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:32,893 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:32,894 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:32,895 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:32,895 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:32,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:32,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:32,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:32,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:32,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:32,895 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:32,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:32,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:32,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:32,896 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:32,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:32,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:32,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:32,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,897 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:32,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:32,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,903 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,903 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:32,903 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:32,903 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:32,903 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:32,903 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:32,903 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:32,903 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:32,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:32,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:32,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:32,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:32,903 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:32,904 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:32,903 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:32,904 DEBUG [Thread-2431] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:32,904 DEBUG [Thread-2431] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:32,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:32,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:32,932 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=495, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/A/9d67004a39cf4c9e904906f4e49d2341 2023-04-09 09:55:32,941 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=495 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/B/f2eb2dc6207d49f9bd568053f6bca1e5 2023-04-09 09:55:33,004 DEBUG [Thread-2431] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:33,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:33,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:33,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:33,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:33,008 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:33,008 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:33,008 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:33,008 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:33,008 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:33,009 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:33,010 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:33,010 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:33,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:33,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:33,010 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:33,010 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:33,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:33,010 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:33,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:33,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:33,011 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:33,011 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:33,011 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:33,011 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:33,011 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:33,011 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:33,011 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:33,011 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:33,011 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:33,012 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. started... 2023-04-09 09:55:33,012 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. as already flushing 2023-04-09 09:55:33,012 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:33,012 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:33,012 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:33,012 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:33,012 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:33,012 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:33,013 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:33,013 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:33,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:33,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:33,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:33,014 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:33,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:33,014 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:33,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:33,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:33,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:33,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:33,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:33,015 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:33,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6debd3e[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:33,015 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:33,016 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:33,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:33,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:33,016 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:33,016 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:33,016 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:33,016 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:33,016 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:33,020 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:33,020 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:33,020 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:33,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:33,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:33,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:33,020 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:33,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:33,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:33,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:33,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:33,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:33,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:33,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:33,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:33,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:33,022 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:33,022 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:33,022 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:33,022 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:33,022 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:33,024 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:33,024 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:33,024 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:33,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:33,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:33,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:33,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:33,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:33,025 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:33,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:33,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:33,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:33,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:33,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:33,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:33,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:33,026 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:33,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:33,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:33,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:33,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:33,030 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:33,030 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:33,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:33,030 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:33,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:33,030 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:33,030 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:33,030 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:33,030 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:33,030 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:33,030 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:33,030 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:33,030 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:33,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:33,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:33,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:33,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:33,032 DEBUG [Thread-2431] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:33,032 DEBUG [Thread-2431] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:33,116 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:33,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32856 deadline: 1681034193110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:33,116 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:33,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32882 deadline: 1681034193111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:33,132 DEBUG [Thread-2431] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:33,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:33,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:33,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:33,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:33,135 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:33,135 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:33,135 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:33,135 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:33,136 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:33,136 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:33,137 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:33,137 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:33,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:33,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:33,137 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:33,137 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:33,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:33,138 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:33,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:33,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:33,138 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:33,138 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:33,138 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:33,138 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:33,139 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:33,139 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:33,139 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:33,139 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:33,139 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:33,139 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. started... 2023-04-09 09:55:33,139 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. as already flushing 2023-04-09 09:55:33,139 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:33,139 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:33,139 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:33,139 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:33,139 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:33,139 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:33,141 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:33,141 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:33,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:33,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:33,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:33,142 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:33,142 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:33,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:33,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:33,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:33,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:33,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:33,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:33,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2786cd92[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:33,143 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:33,143 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:33,144 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:33,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:33,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:33,144 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:33,144 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:33,144 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:33,144 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:33,144 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:33,153 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:33,153 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:33,153 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:33,153 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:33,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:33,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:33,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:33,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:33,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:33,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:33,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:33,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:33,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:33,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:33,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:33,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:33,156 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:33,156 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:33,156 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:33,156 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:33,156 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:33,159 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:33,159 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:33,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:33,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:33,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:33,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:33,159 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:33,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:33,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:33,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:33,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:33,159 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:33,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:33,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:33,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:33,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:33,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:33,161 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:33,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:33,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:33,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:33,162 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:33,162 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:33,162 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:33,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:33,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:33,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:33,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:33,163 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:33,163 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:33,163 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:33,163 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:33,163 DEBUG [Thread-2431] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:33,163 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:33,163 DEBUG [Thread-2431] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:33,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:33,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:33,163 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:33,163 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:33,177 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:33,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32788 deadline: 1681034193177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:33,184 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:33,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32870 deadline: 1681034193178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:33,184 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:33,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32772 deadline: 1681034193179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:33,263 DEBUG [Thread-2431] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:33,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:33,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:33,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:33,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:33,266 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:33,266 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:33,266 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:33,266 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:33,267 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:33,267 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:33,268 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:33,268 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:33,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:33,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:33,268 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:33,268 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:33,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:33,269 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:33,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:33,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:33,269 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:33,269 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:33,269 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:33,269 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:33,269 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:33,269 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:33,269 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:33,269 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:33,270 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:33,270 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. started... 2023-04-09 09:55:33,270 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. as already flushing 2023-04-09 09:55:33,270 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:33,270 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:33,270 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:33,270 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:33,270 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:33,270 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:33,271 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:33,271 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:33,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:33,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:33,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:33,271 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:33,271 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:33,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:33,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:33,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:33,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:33,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:33,273 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:33,273 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d806bf6[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:33,273 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:33,273 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:33,274 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:33,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:33,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:33,275 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:33,275 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:33,275 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:33,275 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:33,275 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:33,276 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:33,276 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:33,276 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:33,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:33,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:33,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:33,276 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:33,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:33,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:33,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:33,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:33,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:33,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:33,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:33,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:33,278 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:33,278 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:33,278 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:33,278 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:33,278 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:33,278 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:33,280 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:33,280 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:33,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:33,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:33,280 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:33,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:33,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:33,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:33,280 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:33,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:33,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:33,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:33,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:33,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:33,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:33,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:33,281 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:33,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:33,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:33,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:33,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:33,283 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:33,283 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:33,283 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:33,283 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:33,283 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:33,283 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:33,283 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:33,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:33,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:33,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:33,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:33,283 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:33,283 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:33,283 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:33,283 DEBUG [Thread-2431] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:33,284 DEBUG [Thread-2431] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:33,284 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:33,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:33,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:33,354 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=495 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/C/4286b516fcce4c6ea09d252fd6fed361 2023-04-09 09:55:33,358 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/A/9d67004a39cf4c9e904906f4e49d2341 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/9d67004a39cf4c9e904906f4e49d2341 2023-04-09 09:55:33,361 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/9d67004a39cf4c9e904906f4e49d2341, entries=150, sequenceid=495, filesize=25.4 K 2023-04-09 09:55:33,362 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/B/f2eb2dc6207d49f9bd568053f6bca1e5 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/f2eb2dc6207d49f9bd568053f6bca1e5 2023-04-09 09:55:33,365 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/f2eb2dc6207d49f9bd568053f6bca1e5, entries=150, sequenceid=495, filesize=11.9 K 2023-04-09 09:55:33,366 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/C/4286b516fcce4c6ea09d252fd6fed361 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/4286b516fcce4c6ea09d252fd6fed361 2023-04-09 09:55:33,368 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/4286b516fcce4c6ea09d252fd6fed361, entries=150, sequenceid=495, filesize=11.9 K 2023-04-09 09:55:33,369 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for bf27154b49f0831beaa90741ce575bc9 in 864ms, sequenceid=495, compaction requested=true 2023-04-09 09:55:33,369 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bf27154b49f0831beaa90741ce575bc9: 2023-04-09 09:55:33,369 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:55:33,369 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-09 09:55:33,369 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:55:33,369 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-09 09:55:33,369 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:55:33,371 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 104934 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-09 09:55:33,371 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1912): bf27154b49f0831beaa90741ce575bc9/A is initiating minor compaction (all files) 2023-04-09 09:55:33,371 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bf27154b49f0831beaa90741ce575bc9/A in TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:33,371 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/eaf3a41bda4b44748367956245961c2a, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/e1f848d9dd494b3c9466f10c56b80d2e, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/a1a77ec971704dcaae9592c69ec90eb9, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/9d67004a39cf4c9e904906f4e49d2341] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp, totalSize=102.5 K 2023-04-09 09:55:33,371 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49494 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-09 09:55:33,371 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): bf27154b49f0831beaa90741ce575bc9/B is initiating minor compaction (all files) 2023-04-09 09:55:33,371 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting eaf3a41bda4b44748367956245961c2a, keycount=150, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=446, earliestPutTs=1681034129043 2023-04-09 09:55:33,372 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bf27154b49f0831beaa90741ce575bc9/B in TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:33,372 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/39dae982f0034f888603ec8aad6c4249, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/f6e68e8daa704354a4331786e188bdf0, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/a4542f1050774377973fb7c67c2092b6, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/f2eb2dc6207d49f9bd568053f6bca1e5] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp, totalSize=48.3 K 2023-04-09 09:55:33,372 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting e1f848d9dd494b3c9466f10c56b80d2e, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=459, earliestPutTs=1681034130179 2023-04-09 09:55:33,372 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 39dae982f0034f888603ec8aad6c4249, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=446, earliestPutTs=1681034129043 2023-04-09 09:55:33,372 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting a1a77ec971704dcaae9592c69ec90eb9, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=481, earliestPutTs=1681034130331 2023-04-09 09:55:33,372 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting f6e68e8daa704354a4331786e188bdf0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=459, earliestPutTs=1681034130179 2023-04-09 09:55:33,373 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 9d67004a39cf4c9e904906f4e49d2341, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=495, earliestPutTs=1681034132489 2023-04-09 09:55:33,373 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting a4542f1050774377973fb7c67c2092b6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=481, earliestPutTs=1681034130331 2023-04-09 09:55:33,373 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting f2eb2dc6207d49f9bd568053f6bca1e5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=495, earliestPutTs=1681034132489 2023-04-09 09:55:33,384 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): bf27154b49f0831beaa90741ce575bc9#B#compaction#406 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:55:33,388 DEBUG [Thread-2431] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:33,388 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] throttle.PressureAwareThroughputController(145): bf27154b49f0831beaa90741ce575bc9#A#compaction#405 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:55:33,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:33,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:33,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:33,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:33,404 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:33,404 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:33,404 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:33,404 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:33,404 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:33,404 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:33,407 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:33,407 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:33,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:33,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:33,407 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:33,407 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:33,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:33,408 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:33,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:33,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:33,408 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:33,408 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:33,408 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:33,408 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:33,408 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:33,408 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:33,409 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:33,409 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:33,409 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:33,409 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. started... 2023-04-09 09:55:33,409 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bf27154b49f0831beaa90741ce575bc9 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-09 09:55:33,409 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf27154b49f0831beaa90741ce575bc9, store=A 2023-04-09 09:55:33,409 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:33,409 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf27154b49f0831beaa90741ce575bc9, store=B 2023-04-09 09:55:33,409 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:33,409 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf27154b49f0831beaa90741ce575bc9, store=C 2023-04-09 09:55:33,409 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:33,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on bf27154b49f0831beaa90741ce575bc9 2023-04-09 09:55:33,620 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. as already flushing 2023-04-09 09:55:33,664 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:33,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32856 deadline: 1681034193657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:33,665 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:33,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32882 deadline: 1681034193658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:33,680 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:33,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32788 deadline: 1681034193679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:33,693 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:33,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32772 deadline: 1681034193688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:33,694 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:33,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:32870 deadline: 1681034193689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:33,766 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:33,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32856 deadline: 1681034193765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:33,766 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:33,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32882 deadline: 1681034193765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:33,794 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/B/6099eb27a79b4244a2ffde93784a370d as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/6099eb27a79b4244a2ffde93784a370d 2023-04-09 09:55:33,798 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in bf27154b49f0831beaa90741ce575bc9/B of bf27154b49f0831beaa90741ce575bc9 into 6099eb27a79b4244a2ffde93784a370d(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:55:33,798 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bf27154b49f0831beaa90741ce575bc9: 2023-04-09 09:55:33,798 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9., storeName=bf27154b49f0831beaa90741ce575bc9/B, priority=12, startTime=1681034133369; duration=0sec 2023-04-09 09:55:33,798 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:55:33,798 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-09 09:55:33,799 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49494 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-09 09:55:33,799 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): bf27154b49f0831beaa90741ce575bc9/C is initiating minor compaction (all files) 2023-04-09 09:55:33,799 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bf27154b49f0831beaa90741ce575bc9/C in TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:33,800 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/1b96b68a8c064b63aecee8c4fd917123, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/ca84d2214aea419e9a75df16b769dc83, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/774b1a0a496b4f3c8c2a310478cc074a, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/4286b516fcce4c6ea09d252fd6fed361] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp, totalSize=48.3 K 2023-04-09 09:55:33,800 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 1b96b68a8c064b63aecee8c4fd917123, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=446, earliestPutTs=1681034129043 2023-04-09 09:55:33,800 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting ca84d2214aea419e9a75df16b769dc83, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=459, earliestPutTs=1681034130179 2023-04-09 09:55:33,801 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 774b1a0a496b4f3c8c2a310478cc074a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=481, earliestPutTs=1681034130331 2023-04-09 09:55:33,801 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 4286b516fcce4c6ea09d252fd6fed361, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=495, earliestPutTs=1681034132489 2023-04-09 09:55:33,808 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): bf27154b49f0831beaa90741ce575bc9#C#compaction#408 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:55:33,823 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/A/a7ab2f8c44d4463385b0c5bbe2a58a5f as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/a7ab2f8c44d4463385b0c5bbe2a58a5f 2023-04-09 09:55:33,824 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304094f22df39a47f4b33a6bd489f8e6d477d to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304094f22df39a47f4b33a6bd489f8e6d477d 2023-04-09 09:55:33,830 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=513, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/A/9f24952425c947cc9518b6c4663f2a49 2023-04-09 09:55:33,831 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in bf27154b49f0831beaa90741ce575bc9/A of bf27154b49f0831beaa90741ce575bc9 into a7ab2f8c44d4463385b0c5bbe2a58a5f(size=26.3 K), total size for store is 26.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:55:33,831 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bf27154b49f0831beaa90741ce575bc9: 2023-04-09 09:55:33,831 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9., storeName=bf27154b49f0831beaa90741ce575bc9/A, priority=12, startTime=1681034133369; duration=0sec 2023-04-09 09:55:33,831 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:55:33,969 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:33,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32856 deadline: 1681034193967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:33,972 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:33,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32882 deadline: 1681034193968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:34,222 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/C/2c8739d36db144eb8f92a2f8c9a84225 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/2c8739d36db144eb8f92a2f8c9a84225 2023-04-09 09:55:34,227 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in bf27154b49f0831beaa90741ce575bc9/C of bf27154b49f0831beaa90741ce575bc9 into 2c8739d36db144eb8f92a2f8c9a84225(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:55:34,227 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bf27154b49f0831beaa90741ce575bc9: 2023-04-09 09:55:34,227 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9., storeName=bf27154b49f0831beaa90741ce575bc9/C, priority=12, startTime=1681034133369; duration=0sec 2023-04-09 09:55:34,227 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:55:34,232 DEBUG [Thread-2432] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x06a0df34 to 127.0.0.1:57816 2023-04-09 09:55:34,232 DEBUG [Thread-2432] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-09 09:55:34,235 DEBUG [Thread-2436] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x21128196 to 127.0.0.1:57816 2023-04-09 09:55:34,235 DEBUG [Thread-2436] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-09 09:55:34,236 DEBUG [Thread-2434] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x10ee5f06 to 127.0.0.1:57816 2023-04-09 09:55:34,238 DEBUG [Thread-2434] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-09 09:55:34,238 DEBUG [Thread-2440] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x711ebaf2 to 127.0.0.1:57816 2023-04-09 09:55:34,238 DEBUG [Thread-2440] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-09 09:55:34,239 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=513 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/B/193ee85c913146248d1130c11a42af70 2023-04-09 09:55:34,241 DEBUG [Thread-2438] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5f9e86d9 to 127.0.0.1:57816 2023-04-09 09:55:34,242 DEBUG [Thread-2438] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-09 09:55:34,271 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:34,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32856 deadline: 1681034194271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:34,275 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:34,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:32882 deadline: 1681034194275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:34,647 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=513 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/C/c6bab92c808e4733ad63b24e52971add 2023-04-09 09:55:34,651 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/A/9f24952425c947cc9518b6c4663f2a49 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/9f24952425c947cc9518b6c4663f2a49 2023-04-09 09:55:34,654 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/9f24952425c947cc9518b6c4663f2a49, entries=150, sequenceid=513, filesize=25.4 K 2023-04-09 09:55:34,654 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/B/193ee85c913146248d1130c11a42af70 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/193ee85c913146248d1130c11a42af70 2023-04-09 09:55:34,657 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/193ee85c913146248d1130c11a42af70, entries=150, sequenceid=513, filesize=11.9 K 2023-04-09 09:55:34,658 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/C/c6bab92c808e4733ad63b24e52971add as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/c6bab92c808e4733ad63b24e52971add 2023-04-09 09:55:34,661 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/c6bab92c808e4733ad63b24e52971add, entries=150, sequenceid=513, filesize=11.9 K 2023-04-09 09:55:34,661 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for bf27154b49f0831beaa90741ce575bc9 in 1252ms, sequenceid=513, compaction requested=false 2023-04-09 09:55:34,661 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bf27154b49f0831beaa90741ce575bc9: 2023-04-09 09:55:34,661 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:34,662 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:34,662 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:34,662 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:34,662 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:34,662 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:34,663 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:34,663 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:34,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:34,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:34,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:34,663 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:34,663 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:34,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:34,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:34,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:34,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:34,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:34,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:34,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3bfc2199[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:34,665 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:34,665 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:34,666 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:34,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:34,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:34,666 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:34,666 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:34,666 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:34,666 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:34,666 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:34,668 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:34,668 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:34,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:34,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:34,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:34,668 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:34,668 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:34,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:34,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:34,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:34,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:34,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:34,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:34,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:34,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:34,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:34,670 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:34,670 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:34,670 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:34,670 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:34,670 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:34,671 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:34,671 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:34,671 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:34,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:34,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:34,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:34,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:34,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:34,672 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:34,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:34,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:34,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:34,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:34,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:34,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:34,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:34,673 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:34,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:34,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:34,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:34,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:34,675 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:34,675 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:34,675 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:34,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:34,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:34,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:34,675 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:34,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:34,675 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:34,676 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:34,675 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:34,676 DEBUG [Thread-2431] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:34,676 DEBUG [Thread-2431] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:34,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:34,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:34,676 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:34,676 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:34,676 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:34,676 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:34,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on bf27154b49f0831beaa90741ce575bc9 2023-04-09 09:55:34,683 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bf27154b49f0831beaa90741ce575bc9 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-09 09:55:34,683 DEBUG [Thread-2423] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0bb27fd4 to 127.0.0.1:57816 2023-04-09 09:55:34,683 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf27154b49f0831beaa90741ce575bc9, store=A 2023-04-09 09:55:34,683 DEBUG [Thread-2423] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-09 09:55:34,683 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:34,683 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf27154b49f0831beaa90741ce575bc9, store=B 2023-04-09 09:55:34,683 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:34,683 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf27154b49f0831beaa90741ce575bc9, store=C 2023-04-09 09:55:34,683 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:34,692 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230409ed8f388c2d0640949901fc79069928b9 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409ed8f388c2d0640949901fc79069928b9 2023-04-09 09:55:34,696 DEBUG [Thread-2425] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x445919fd to 127.0.0.1:57816 2023-04-09 09:55:34,696 DEBUG [Thread-2425] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-09 09:55:34,696 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=531, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/A/4e64af2de9474a6190cf3ab3cc8620a8 2023-04-09 09:55:34,701 DEBUG [Thread-2427] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4d010713 to 127.0.0.1:57816 2023-04-09 09:55:34,702 DEBUG [Thread-2427] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-09 09:55:34,773 DEBUG [Thread-2429] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x69100738 to 127.0.0.1:57816 2023-04-09 09:55:34,773 DEBUG [Thread-2429] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-09 09:55:34,776 DEBUG [Thread-2431] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:34,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:34,778 DEBUG [Thread-2421] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6434ce29 to 127.0.0.1:57816 2023-04-09 09:55:34,779 DEBUG [Thread-2421] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-09 09:55:34,779 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-09 09:55:34,779 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(394): wrote 139 2023-04-09 09:55:34,779 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(394): wrote 143 2023-04-09 09:55:34,779 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(394): wrote 23 2023-04-09 09:55:34,779 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(394): wrote 17 2023-04-09 09:55:34,779 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(394): wrote 122 2023-04-09 09:55:34,779 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-09 09:55:34,779 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-09 09:55:34,779 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(402): scanned 1985 2023-04-09 09:55:34,779 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(403): verified 5955 rows 2023-04-09 09:55:34,779 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(402): scanned 1979 2023-04-09 09:55:34,779 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(403): verified 5937 rows 2023-04-09 09:55:34,779 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(402): scanned 1985 2023-04-09 09:55:34,779 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(403): verified 5955 rows 2023-04-09 09:55:34,779 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(402): scanned 1956 2023-04-09 09:55:34,779 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(403): verified 5861 rows 2023-04-09 09:55:34,779 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(402): scanned 1982 2023-04-09 09:55:34,779 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(403): verified 5946 rows 2023-04-09 09:55:34,779 INFO [Listener at localhost/44563] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-09 09:55:34,780 DEBUG [Listener at localhost/44563] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1b77d6d7 to 127.0.0.1:57816 2023-04-09 09:55:34,780 DEBUG [Listener at localhost/44563] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-09 09:55:34,784 INFO [Listener at localhost/44563] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-09 09:55:34,784 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-09 09:55:34,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure2.ProcedureExecutor(1029): Stored pid=43, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-09 09:55:34,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-09 09:55:34,787 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681034134787"}]},"ts":"1681034134787"} 2023-04-09 09:55:34,788 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-09 09:55:34,789 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-09 09:55:34,790 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=bf27154b49f0831beaa90741ce575bc9, UNASSIGN}] 2023-04-09 09:55:34,791 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=bf27154b49f0831beaa90741ce575bc9, UNASSIGN 2023-04-09 09:55:34,791 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=bf27154b49f0831beaa90741ce575bc9, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:34,792 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681034134791"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681034134791"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681034134791"}]},"ts":"1681034134791"} 2023-04-09 09:55:34,792 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE; CloseRegionProcedure bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544}] 2023-04-09 09:55:34,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-09 09:55:34,944 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close bf27154b49f0831beaa90741ce575bc9 2023-04-09 09:55:34,944 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing bf27154b49f0831beaa90741ce575bc9, disabling compactions & flushes 2023-04-09 09:55:34,944 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:35,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-09 09:55:35,104 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=531 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/B/ffbf8c34663749b3be776f4b5999656d 2023-04-09 09:55:35,113 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=531 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/C/905c552657f44de59fd1d11433108e8b 2023-04-09 09:55:35,116 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/A/4e64af2de9474a6190cf3ab3cc8620a8 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/4e64af2de9474a6190cf3ab3cc8620a8 2023-04-09 09:55:35,119 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/4e64af2de9474a6190cf3ab3cc8620a8, entries=150, sequenceid=531, filesize=25.4 K 2023-04-09 09:55:35,119 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/B/ffbf8c34663749b3be776f4b5999656d as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/ffbf8c34663749b3be776f4b5999656d 2023-04-09 09:55:35,122 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/ffbf8c34663749b3be776f4b5999656d, entries=150, sequenceid=531, filesize=11.9 K 2023-04-09 09:55:35,123 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/C/905c552657f44de59fd1d11433108e8b as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/905c552657f44de59fd1d11433108e8b 2023-04-09 09:55:35,125 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/905c552657f44de59fd1d11433108e8b, entries=150, sequenceid=531, filesize=11.9 K 2023-04-09 09:55:35,126 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=26.84 KB/27480 for bf27154b49f0831beaa90741ce575bc9 in 443ms, sequenceid=531, compaction requested=true 2023-04-09 09:55:35,126 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bf27154b49f0831beaa90741ce575bc9: 2023-04-09 09:55:35,126 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:35,126 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. because compaction request was cancelled 2023-04-09 09:55:35,126 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:35,126 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:55:35,126 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. after waiting 0 ms 2023-04-09 09:55:35,126 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:55:35,126 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. because compaction request was cancelled 2023-04-09 09:55:35,126 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:55:35,126 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. because compaction request was cancelled 2023-04-09 09:55:35,126 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:35,127 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing bf27154b49f0831beaa90741ce575bc9 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-09 09:55:35,127 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf27154b49f0831beaa90741ce575bc9, store=A 2023-04-09 09:55:35,127 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:35,127 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf27154b49f0831beaa90741ce575bc9, store=B 2023-04-09 09:55:35,127 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:35,127 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bf27154b49f0831beaa90741ce575bc9, store=C 2023-04-09 09:55:35,127 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:35,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-09 09:55:35,536 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230409a2c8f196f23d437989ffd86d413054ad to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409a2c8f196f23d437989ffd86d413054ad 2023-04-09 09:55:35,540 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=538, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/A/27507a73d52a446784dd3a9bdb48261c 2023-04-09 09:55:35,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-09 09:55:35,950 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=538 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/B/35ce084f3f9242439438551d05fa0075 2023-04-09 09:55:36,358 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=538 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/C/2b4c9d4705754e55aa26789550ec71e7 2023-04-09 09:55:36,361 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/A/27507a73d52a446784dd3a9bdb48261c as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/27507a73d52a446784dd3a9bdb48261c 2023-04-09 09:55:36,363 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/27507a73d52a446784dd3a9bdb48261c, entries=150, sequenceid=538, filesize=25.4 K 2023-04-09 09:55:36,364 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/B/35ce084f3f9242439438551d05fa0075 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/35ce084f3f9242439438551d05fa0075 2023-04-09 09:55:36,367 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/35ce084f3f9242439438551d05fa0075, entries=150, sequenceid=538, filesize=11.9 K 2023-04-09 09:55:36,367 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/.tmp/C/2b4c9d4705754e55aa26789550ec71e7 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/2b4c9d4705754e55aa26789550ec71e7 2023-04-09 09:55:36,370 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/2b4c9d4705754e55aa26789550ec71e7, entries=150, sequenceid=538, filesize=11.9 K 2023-04-09 09:55:36,370 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for bf27154b49f0831beaa90741ce575bc9 in 1244ms, sequenceid=538, compaction requested=true 2023-04-09 09:55:36,393 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/6414cc9aef4a4bb18f9a7cc6cc04fd59, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/7d245810b3aa4e10a7b61f5caa0e04c8, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/27f622741ce54d2c85f56052891edd91, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/07f0337b87d447b0b75cb75da29dacee, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/9a866099139546a1986cc0f5e4877c3c, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/f19f8e87c9d441a6814e7e412dbbda24, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/7387c8f2f2694f2b8583c1145a72f45c, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/e0cbce6fc27c484bbc7a8b518327e484, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/a64933e6e15d43b39e74cfc55557e4ad, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/cd46d9c1278c4db296846d0bdadbbf98, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/77efd3581c054e84a8328fe75cb55b1c, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/73d8c0c0dddd42cba1445198b67b32ec, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/e5e307222e684b94b625fc9b70cb0bd7, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/7512c91429784267933243c027150373, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/c25885a6d5d14e2d894d1c6a7738bec8, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/7e55f0674f8c4055bff695f1c2c03053, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/aed07d1272364b61bc9b6c1224c1c147, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/97461705218e4a728203ab23f705a606, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/980054979d6247109a79cff71f6cc19e, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/f1f0c8d6c2f54f0fbc8c6222c5ce3945, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/3d7337c77ae7439391a3cf5cc90280f4, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/588ab4fdf564493e9acbaeb51b95896c, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/eaf3a41bda4b44748367956245961c2a, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/e1f848d9dd494b3c9466f10c56b80d2e, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/a1a77ec971704dcaae9592c69ec90eb9, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/9d67004a39cf4c9e904906f4e49d2341] to archive 2023-04-09 09:55:36,394 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-09 09:55:36,395 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/6414cc9aef4a4bb18f9a7cc6cc04fd59 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/6414cc9aef4a4bb18f9a7cc6cc04fd59 2023-04-09 09:55:36,396 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/7d245810b3aa4e10a7b61f5caa0e04c8 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/7d245810b3aa4e10a7b61f5caa0e04c8 2023-04-09 09:55:36,397 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/27f622741ce54d2c85f56052891edd91 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/27f622741ce54d2c85f56052891edd91 2023-04-09 09:55:36,398 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/07f0337b87d447b0b75cb75da29dacee to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/07f0337b87d447b0b75cb75da29dacee 2023-04-09 09:55:36,399 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/9a866099139546a1986cc0f5e4877c3c to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/9a866099139546a1986cc0f5e4877c3c 2023-04-09 09:55:36,400 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/f19f8e87c9d441a6814e7e412dbbda24 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/f19f8e87c9d441a6814e7e412dbbda24 2023-04-09 09:55:36,400 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/7387c8f2f2694f2b8583c1145a72f45c to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/7387c8f2f2694f2b8583c1145a72f45c 2023-04-09 09:55:36,401 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/e0cbce6fc27c484bbc7a8b518327e484 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/e0cbce6fc27c484bbc7a8b518327e484 2023-04-09 09:55:36,401 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/a64933e6e15d43b39e74cfc55557e4ad to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/a64933e6e15d43b39e74cfc55557e4ad 2023-04-09 09:55:36,402 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/cd46d9c1278c4db296846d0bdadbbf98 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/cd46d9c1278c4db296846d0bdadbbf98 2023-04-09 09:55:36,403 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/77efd3581c054e84a8328fe75cb55b1c to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/77efd3581c054e84a8328fe75cb55b1c 2023-04-09 09:55:36,403 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/73d8c0c0dddd42cba1445198b67b32ec to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/73d8c0c0dddd42cba1445198b67b32ec 2023-04-09 09:55:36,404 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/e5e307222e684b94b625fc9b70cb0bd7 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/e5e307222e684b94b625fc9b70cb0bd7 2023-04-09 09:55:36,404 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/7512c91429784267933243c027150373 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/7512c91429784267933243c027150373 2023-04-09 09:55:36,405 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/c25885a6d5d14e2d894d1c6a7738bec8 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/c25885a6d5d14e2d894d1c6a7738bec8 2023-04-09 09:55:36,406 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/7e55f0674f8c4055bff695f1c2c03053 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/7e55f0674f8c4055bff695f1c2c03053 2023-04-09 09:55:36,407 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/aed07d1272364b61bc9b6c1224c1c147 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/aed07d1272364b61bc9b6c1224c1c147 2023-04-09 09:55:36,408 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/97461705218e4a728203ab23f705a606 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/97461705218e4a728203ab23f705a606 2023-04-09 09:55:36,409 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/980054979d6247109a79cff71f6cc19e to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/980054979d6247109a79cff71f6cc19e 2023-04-09 09:55:36,410 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/f1f0c8d6c2f54f0fbc8c6222c5ce3945 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/f1f0c8d6c2f54f0fbc8c6222c5ce3945 2023-04-09 09:55:36,410 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/3d7337c77ae7439391a3cf5cc90280f4 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/3d7337c77ae7439391a3cf5cc90280f4 2023-04-09 09:55:36,411 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/588ab4fdf564493e9acbaeb51b95896c to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/588ab4fdf564493e9acbaeb51b95896c 2023-04-09 09:55:36,412 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/eaf3a41bda4b44748367956245961c2a to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/eaf3a41bda4b44748367956245961c2a 2023-04-09 09:55:36,413 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/e1f848d9dd494b3c9466f10c56b80d2e to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/e1f848d9dd494b3c9466f10c56b80d2e 2023-04-09 09:55:36,414 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/a1a77ec971704dcaae9592c69ec90eb9 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/a1a77ec971704dcaae9592c69ec90eb9 2023-04-09 09:55:36,415 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/9d67004a39cf4c9e904906f4e49d2341 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/9d67004a39cf4c9e904906f4e49d2341 2023-04-09 09:55:36,418 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/0f73b8ad946d4199b91fddf0bc874460, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/c0aa63628b4548bba9707d3173ed0b3f, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/ef311de467a94c84bb1bc30e5b4a83be, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/d6a39d3d76dd49cbac4b5fe71c4bc827, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/9ac2a805efa14c969071a482eb009c65, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/1e70a4e5eec94a2cbd07796c7d20a556, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/173860f44f104ca3a42cfc17f917511f, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/f0829384229a4f94841c271a93b58ca3, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/9169639d247a48a981b1b7a471bdca44, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/499248fcbec64e408ac9c394328633a0, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/b1c649bb260d4bd6a8fa7949fe53b4f3, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/00767cdbf95748e8b61ccc0f2b3e3a20, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/f0d4f99efe0449e58ab816307ca3b37c, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/b299072101ea486a9e0e7220c9482f9e, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/75b37fef0ce8492180570b1a1cdd336d, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/4d47557aa244452d8b6a00a7a9854c7f, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/9bbef30011294f5186c542b0312d593d, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/129d769e7e83484391fac9f78a9c145a, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/8c0e97f8103b4cd68c9782c422bc335b, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/e9b11377a989482a9f357714bb4854f1, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/b33dc6b1b4a845d5b560a653ca07cb84, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/39dae982f0034f888603ec8aad6c4249, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/7ef6f5ddbf5f43fdb9650f7e70170dcf, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/f6e68e8daa704354a4331786e188bdf0, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/a4542f1050774377973fb7c67c2092b6, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/f2eb2dc6207d49f9bd568053f6bca1e5] to archive 2023-04-09 09:55:36,418 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-09 09:55:36,420 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/0f73b8ad946d4199b91fddf0bc874460 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/0f73b8ad946d4199b91fddf0bc874460 2023-04-09 09:55:36,421 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/c0aa63628b4548bba9707d3173ed0b3f to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/c0aa63628b4548bba9707d3173ed0b3f 2023-04-09 09:55:36,422 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/ef311de467a94c84bb1bc30e5b4a83be to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/ef311de467a94c84bb1bc30e5b4a83be 2023-04-09 09:55:36,423 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/d6a39d3d76dd49cbac4b5fe71c4bc827 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/d6a39d3d76dd49cbac4b5fe71c4bc827 2023-04-09 09:55:36,424 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/9ac2a805efa14c969071a482eb009c65 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/9ac2a805efa14c969071a482eb009c65 2023-04-09 09:55:36,425 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/1e70a4e5eec94a2cbd07796c7d20a556 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/1e70a4e5eec94a2cbd07796c7d20a556 2023-04-09 09:55:36,426 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/173860f44f104ca3a42cfc17f917511f to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/173860f44f104ca3a42cfc17f917511f 2023-04-09 09:55:36,427 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/f0829384229a4f94841c271a93b58ca3 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/f0829384229a4f94841c271a93b58ca3 2023-04-09 09:55:36,428 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/9169639d247a48a981b1b7a471bdca44 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/9169639d247a48a981b1b7a471bdca44 2023-04-09 09:55:36,429 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/499248fcbec64e408ac9c394328633a0 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/499248fcbec64e408ac9c394328633a0 2023-04-09 09:55:36,430 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/b1c649bb260d4bd6a8fa7949fe53b4f3 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/b1c649bb260d4bd6a8fa7949fe53b4f3 2023-04-09 09:55:36,431 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/00767cdbf95748e8b61ccc0f2b3e3a20 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/00767cdbf95748e8b61ccc0f2b3e3a20 2023-04-09 09:55:36,432 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/f0d4f99efe0449e58ab816307ca3b37c to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/f0d4f99efe0449e58ab816307ca3b37c 2023-04-09 09:55:36,433 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/b299072101ea486a9e0e7220c9482f9e to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/b299072101ea486a9e0e7220c9482f9e 2023-04-09 09:55:36,433 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/75b37fef0ce8492180570b1a1cdd336d to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/75b37fef0ce8492180570b1a1cdd336d 2023-04-09 09:55:36,434 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/4d47557aa244452d8b6a00a7a9854c7f to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/4d47557aa244452d8b6a00a7a9854c7f 2023-04-09 09:55:36,435 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/9bbef30011294f5186c542b0312d593d to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/9bbef30011294f5186c542b0312d593d 2023-04-09 09:55:36,436 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/129d769e7e83484391fac9f78a9c145a to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/129d769e7e83484391fac9f78a9c145a 2023-04-09 09:55:36,437 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/8c0e97f8103b4cd68c9782c422bc335b to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/8c0e97f8103b4cd68c9782c422bc335b 2023-04-09 09:55:36,437 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/e9b11377a989482a9f357714bb4854f1 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/e9b11377a989482a9f357714bb4854f1 2023-04-09 09:55:36,438 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/b33dc6b1b4a845d5b560a653ca07cb84 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/b33dc6b1b4a845d5b560a653ca07cb84 2023-04-09 09:55:36,439 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/39dae982f0034f888603ec8aad6c4249 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/39dae982f0034f888603ec8aad6c4249 2023-04-09 09:55:36,440 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/7ef6f5ddbf5f43fdb9650f7e70170dcf to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/7ef6f5ddbf5f43fdb9650f7e70170dcf 2023-04-09 09:55:36,440 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/f6e68e8daa704354a4331786e188bdf0 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/f6e68e8daa704354a4331786e188bdf0 2023-04-09 09:55:36,441 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/a4542f1050774377973fb7c67c2092b6 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/a4542f1050774377973fb7c67c2092b6 2023-04-09 09:55:36,442 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/f2eb2dc6207d49f9bd568053f6bca1e5 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/f2eb2dc6207d49f9bd568053f6bca1e5 2023-04-09 09:55:36,444 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/65ccd0efd6894cbc93de631d4f2b993f, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/c3d6fa61659e4dd5a374c671526fbb25, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/22d3b8e59d0443bf9e44a9bfbe2d9778, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/5bd2e8b8bdc34f83a64e17fa582ef1d8, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/655b866eaea04f59b092b653ce7cf38c, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/a14e2ebb4b2643ca84ade277d9bd11e3, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/b7d65a78164240969fdafee199635db2, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/29e6b10df77e46d4b6832898ee7eab53, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/27458d76c0034be3bc60a79ca3306061, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/cef0e2878c60471bb3fc23042940ba10, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/9cf5c68d3fd7490493124dad1a99dbf3, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/3ee5161b43d246ebbf00ff8bf515831f, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/b9cff78f8f5f42a08da671626df1c61e, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/6c68f163b0764b2ebc518f1eb148039b, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/9bc340e4dd5c4eb7a24905b6dd04a105, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/95713831047a4fe0a66caf92bc158a31, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/bf604afbc75445c9b86d4776c849e3e9, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/602443387b5b4e1abab3773aff5b4b9a, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/046383ca20234eb182b5d613f999aedb, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/a89644051cc447d8bf9c0661d7eeedb2, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/80120736f84a456681e912b570cb30b0, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/1b96b68a8c064b63aecee8c4fd917123, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/75ca0c8dcac1435ea55245628925338e, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/ca84d2214aea419e9a75df16b769dc83, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/774b1a0a496b4f3c8c2a310478cc074a, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/4286b516fcce4c6ea09d252fd6fed361] to archive 2023-04-09 09:55:36,445 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-09 09:55:36,446 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/65ccd0efd6894cbc93de631d4f2b993f to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/65ccd0efd6894cbc93de631d4f2b993f 2023-04-09 09:55:36,447 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/c3d6fa61659e4dd5a374c671526fbb25 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/c3d6fa61659e4dd5a374c671526fbb25 2023-04-09 09:55:36,448 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/22d3b8e59d0443bf9e44a9bfbe2d9778 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/22d3b8e59d0443bf9e44a9bfbe2d9778 2023-04-09 09:55:36,448 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/5bd2e8b8bdc34f83a64e17fa582ef1d8 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/5bd2e8b8bdc34f83a64e17fa582ef1d8 2023-04-09 09:55:36,449 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/655b866eaea04f59b092b653ce7cf38c to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/655b866eaea04f59b092b653ce7cf38c 2023-04-09 09:55:36,450 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/a14e2ebb4b2643ca84ade277d9bd11e3 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/a14e2ebb4b2643ca84ade277d9bd11e3 2023-04-09 09:55:36,451 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/b7d65a78164240969fdafee199635db2 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/b7d65a78164240969fdafee199635db2 2023-04-09 09:55:36,452 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/29e6b10df77e46d4b6832898ee7eab53 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/29e6b10df77e46d4b6832898ee7eab53 2023-04-09 09:55:36,453 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/27458d76c0034be3bc60a79ca3306061 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/27458d76c0034be3bc60a79ca3306061 2023-04-09 09:55:36,453 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/cef0e2878c60471bb3fc23042940ba10 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/cef0e2878c60471bb3fc23042940ba10 2023-04-09 09:55:36,454 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/9cf5c68d3fd7490493124dad1a99dbf3 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/9cf5c68d3fd7490493124dad1a99dbf3 2023-04-09 09:55:36,455 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/3ee5161b43d246ebbf00ff8bf515831f to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/3ee5161b43d246ebbf00ff8bf515831f 2023-04-09 09:55:36,456 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/b9cff78f8f5f42a08da671626df1c61e to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/b9cff78f8f5f42a08da671626df1c61e 2023-04-09 09:55:36,456 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/6c68f163b0764b2ebc518f1eb148039b to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/6c68f163b0764b2ebc518f1eb148039b 2023-04-09 09:55:36,457 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/9bc340e4dd5c4eb7a24905b6dd04a105 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/9bc340e4dd5c4eb7a24905b6dd04a105 2023-04-09 09:55:36,458 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/95713831047a4fe0a66caf92bc158a31 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/95713831047a4fe0a66caf92bc158a31 2023-04-09 09:55:36,458 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/bf604afbc75445c9b86d4776c849e3e9 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/bf604afbc75445c9b86d4776c849e3e9 2023-04-09 09:55:36,459 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/602443387b5b4e1abab3773aff5b4b9a to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/602443387b5b4e1abab3773aff5b4b9a 2023-04-09 09:55:36,460 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/046383ca20234eb182b5d613f999aedb to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/046383ca20234eb182b5d613f999aedb 2023-04-09 09:55:36,461 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/a89644051cc447d8bf9c0661d7eeedb2 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/a89644051cc447d8bf9c0661d7eeedb2 2023-04-09 09:55:36,461 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/80120736f84a456681e912b570cb30b0 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/80120736f84a456681e912b570cb30b0 2023-04-09 09:55:36,462 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/1b96b68a8c064b63aecee8c4fd917123 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/1b96b68a8c064b63aecee8c4fd917123 2023-04-09 09:55:36,463 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/75ca0c8dcac1435ea55245628925338e to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/75ca0c8dcac1435ea55245628925338e 2023-04-09 09:55:36,464 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/ca84d2214aea419e9a75df16b769dc83 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/ca84d2214aea419e9a75df16b769dc83 2023-04-09 09:55:36,464 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/774b1a0a496b4f3c8c2a310478cc074a to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/774b1a0a496b4f3c8c2a310478cc074a 2023-04-09 09:55:36,465 DEBUG [StoreCloser-TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/4286b516fcce4c6ea09d252fd6fed361 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/4286b516fcce4c6ea09d252fd6fed361 2023-04-09 09:55:36,469 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/recovered.edits/541.seqid, newMaxSeqId=541, maxSeqId=4 2023-04-09 09:55:36,470 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9. 2023-04-09 09:55:36,470 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for bf27154b49f0831beaa90741ce575bc9: 2023-04-09 09:55:36,471 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed bf27154b49f0831beaa90741ce575bc9 2023-04-09 09:55:36,471 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=bf27154b49f0831beaa90741ce575bc9, regionState=CLOSED 2023-04-09 09:55:36,471 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681034136471"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681034136471"}]},"ts":"1681034136471"} 2023-04-09 09:55:36,473 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=44 2023-04-09 09:55:36,473 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=44, state=SUCCESS; CloseRegionProcedure bf27154b49f0831beaa90741ce575bc9, server=jenkins-hbase4.apache.org,42467,1681034019544 in 1.6800 sec 2023-04-09 09:55:36,474 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=43 2023-04-09 09:55:36,474 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=43, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=bf27154b49f0831beaa90741ce575bc9, UNASSIGN in 1.6830 sec 2023-04-09 09:55:36,475 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681034136475"}]},"ts":"1681034136475"} 2023-04-09 09:55:36,475 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-09 09:55:36,477 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-09 09:55:36,478 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=43, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.6930 sec 2023-04-09 09:55:36,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-09 09:55:36,892 INFO [Listener at localhost/44563] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 43 completed 2023-04-09 09:55:36,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-09 09:55:36,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure2.ProcedureExecutor(1029): Stored pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-09 09:55:36,894 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-09 09:55:36,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-09 09:55:36,895 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=46, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-09 09:55:36,899 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9 2023-04-09 09:55:36,901 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A, FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B, FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C, FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/recovered.edits] 2023-04-09 09:55:36,903 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/27507a73d52a446784dd3a9bdb48261c to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/27507a73d52a446784dd3a9bdb48261c 2023-04-09 09:55:36,904 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/4e64af2de9474a6190cf3ab3cc8620a8 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/4e64af2de9474a6190cf3ab3cc8620a8 2023-04-09 09:55:36,905 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/9f24952425c947cc9518b6c4663f2a49 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/9f24952425c947cc9518b6c4663f2a49 2023-04-09 09:55:36,906 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/a7ab2f8c44d4463385b0c5bbe2a58a5f to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/A/a7ab2f8c44d4463385b0c5bbe2a58a5f 2023-04-09 09:55:36,908 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/193ee85c913146248d1130c11a42af70 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/193ee85c913146248d1130c11a42af70 2023-04-09 09:55:36,909 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/35ce084f3f9242439438551d05fa0075 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/35ce084f3f9242439438551d05fa0075 2023-04-09 09:55:36,910 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/6099eb27a79b4244a2ffde93784a370d to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/6099eb27a79b4244a2ffde93784a370d 2023-04-09 09:55:36,910 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/ffbf8c34663749b3be776f4b5999656d to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/B/ffbf8c34663749b3be776f4b5999656d 2023-04-09 09:55:36,912 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/2b4c9d4705754e55aa26789550ec71e7 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/2b4c9d4705754e55aa26789550ec71e7 2023-04-09 09:55:36,913 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/2c8739d36db144eb8f92a2f8c9a84225 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/2c8739d36db144eb8f92a2f8c9a84225 2023-04-09 09:55:36,914 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/905c552657f44de59fd1d11433108e8b to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/905c552657f44de59fd1d11433108e8b 2023-04-09 09:55:36,915 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/c6bab92c808e4733ad63b24e52971add to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/C/c6bab92c808e4733ad63b24e52971add 2023-04-09 09:55:36,917 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/recovered.edits/541.seqid to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9/recovered.edits/541.seqid 2023-04-09 09:55:36,918 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/bf27154b49f0831beaa90741ce575bc9 2023-04-09 09:55:36,918 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-09 09:55:36,918 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-09 09:55:36,919 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-09 09:55:36,921 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304090ba359b4f7bb40bc939477e32f5f37e3 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304090ba359b4f7bb40bc939477e32f5f37e3 2023-04-09 09:55:36,921 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409187b9c6d248c4f06b6da43c534690e1a to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409187b9c6d248c4f06b6da43c534690e1a 2023-04-09 09:55:36,922 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304092ed9da273baa44c78f1ff213e9868500 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304092ed9da273baa44c78f1ff213e9868500 2023-04-09 09:55:36,923 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304092f226f3a05aa45729f0b6bc5a69326f0 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304092f226f3a05aa45729f0b6bc5a69326f0 2023-04-09 09:55:36,924 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304093eeae94c400b475795dc90d950346d31 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304093eeae94c400b475795dc90d950346d31 2023-04-09 09:55:36,925 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040940e42f6a39484955bd61ead508186db8 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040940e42f6a39484955bd61ead508186db8 2023-04-09 09:55:36,926 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304094133548ab9af49d68183cf2b4a606b4c to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304094133548ab9af49d68183cf2b4a606b4c 2023-04-09 09:55:36,927 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304094983d4b1685d48b9b8939dcfe04224c3 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304094983d4b1685d48b9b8939dcfe04224c3 2023-04-09 09:55:36,928 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304094f22df39a47f4b33a6bd489f8e6d477d to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304094f22df39a47f4b33a6bd489f8e6d477d 2023-04-09 09:55:36,928 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409537ea8f5b5c04020a4a6ad1a8739be0b to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409537ea8f5b5c04020a4a6ad1a8739be0b 2023-04-09 09:55:36,929 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040958cd78fcf9e649928f69a4cd2e46a4c4 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040958cd78fcf9e649928f69a4cd2e46a4c4 2023-04-09 09:55:36,930 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304096ec4f86818a04afe97cb01151a51c383 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304096ec4f86818a04afe97cb01151a51c383 2023-04-09 09:55:36,931 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409937ce52a963b4e0da2ca3c25bc286089 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409937ce52a963b4e0da2ca3c25bc286089 2023-04-09 09:55:36,932 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409972811e338a74d87a7911c5bef02a268 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409972811e338a74d87a7911c5bef02a268 2023-04-09 09:55:36,933 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409a2c8f196f23d437989ffd86d413054ad to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409a2c8f196f23d437989ffd86d413054ad 2023-04-09 09:55:36,934 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409a5ecae688e414b5780cc878771a83703 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409a5ecae688e414b5780cc878771a83703 2023-04-09 09:55:36,935 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409b610a613a42f4b469b880bd03dc9f9ff to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409b610a613a42f4b469b880bd03dc9f9ff 2023-04-09 09:55:36,936 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409bfb87e24b4384681ab42a7df1062659a to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409bfb87e24b4384681ab42a7df1062659a 2023-04-09 09:55:36,937 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409ce460e1b47084abebbb3fa68bdff5216 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409ce460e1b47084abebbb3fa68bdff5216 2023-04-09 09:55:36,937 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409e901b38d3a05488bb0f4d2603f57c560 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409e901b38d3a05488bb0f4d2603f57c560 2023-04-09 09:55:36,938 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409eafb78edf48745cf881b4edadb531579 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409eafb78edf48745cf881b4edadb531579 2023-04-09 09:55:36,939 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409ed8f388c2d0640949901fc79069928b9 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409ed8f388c2d0640949901fc79069928b9 2023-04-09 09:55:36,940 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409f9924598a25446baa567bdd665634c44 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409f9924598a25446baa567bdd665634c44 2023-04-09 09:55:36,940 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-09 09:55:36,942 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=46, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-09 09:55:36,943 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-09 09:55:36,945 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-09 09:55:36,945 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=46, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-09 09:55:36,945 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-09 09:55:36,945 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681034136945"}]},"ts":"9223372036854775807"} 2023-04-09 09:55:36,947 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-09 09:55:36,947 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => bf27154b49f0831beaa90741ce575bc9, NAME => 'TestAcidGuarantees,,1681034112142.bf27154b49f0831beaa90741ce575bc9.', STARTKEY => '', ENDKEY => ''}] 2023-04-09 09:55:36,947 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-09 09:55:36,947 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681034136947"}]},"ts":"9223372036854775807"} 2023-04-09 09:55:36,948 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-09 09:55:36,953 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=46, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-09 09:55:36,953 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=46, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 61 msec 2023-04-09 09:55:36,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-09 09:55:36,996 INFO [Listener at localhost/44563] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 46 completed 2023-04-09 09:55:37,007 INFO [Listener at localhost/44563] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=230 (was 224) - Thread LEAK? -, OpenFileDescriptor=546 (was 528) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=386 (was 397), ProcessCount=169 (was 169), AvailableMemoryMB=1437 (was 1499) 2023-04-09 09:55:37,018 INFO [Listener at localhost/44563] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=230, OpenFileDescriptor=546, MaxFileDescriptor=60000, SystemLoadAverage=386, ProcessCount=169, AvailableMemoryMB=1437 2023-04-09 09:55:37,020 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-09 09:55:37,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-09 09:55:37,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure2.ProcedureExecutor(1029): Stored pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-09 09:55:37,023 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-09 09:55:37,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 47 2023-04-09 09:55:37,024 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-09 09:55:37,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-09 09:55:37,024 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00 2023-04-09 09:55:37,025 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00 empty. 2023-04-09 09:55:37,025 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00 2023-04-09 09:55:37,025 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-09 09:55:37,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-09 09:55:37,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-09 09:55:37,433 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-09 09:55:37,434 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 2764d7f8155e1e5b0d641eebe063fa00, NAME => 'TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp 2023-04-09 09:55:37,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-09 09:55:37,840 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-09 09:55:37,840 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 2764d7f8155e1e5b0d641eebe063fa00, disabling compactions & flushes 2023-04-09 09:55:37,840 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:37,840 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:37,840 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. after waiting 0 ms 2023-04-09 09:55:37,840 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:37,840 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:37,840 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 2764d7f8155e1e5b0d641eebe063fa00: 2023-04-09 09:55:37,842 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-09 09:55:37,843 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681034137843"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681034137843"}]},"ts":"1681034137843"} 2023-04-09 09:55:37,843 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-09 09:55:37,844 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-09 09:55:37,844 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681034137844"}]},"ts":"1681034137844"} 2023-04-09 09:55:37,845 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-09 09:55:37,847 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2764d7f8155e1e5b0d641eebe063fa00, ASSIGN}] 2023-04-09 09:55:37,848 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2764d7f8155e1e5b0d641eebe063fa00, ASSIGN 2023-04-09 09:55:37,849 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=2764d7f8155e1e5b0d641eebe063fa00, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,42467,1681034019544; forceNewPlan=false, retain=false 2023-04-09 09:55:38,000 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=2764d7f8155e1e5b0d641eebe063fa00, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:38,000 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681034138000"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681034138000"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681034138000"}]},"ts":"1681034138000"} 2023-04-09 09:55:38,001 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=49, ppid=48, state=RUNNABLE; OpenRegionProcedure 2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544}] 2023-04-09 09:55:38,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-09 09:55:38,156 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:38,156 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 2764d7f8155e1e5b0d641eebe063fa00, NAME => 'TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.', STARTKEY => '', ENDKEY => ''} 2023-04-09 09:55:38,156 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 2764d7f8155e1e5b0d641eebe063fa00 2023-04-09 09:55:38,156 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-09 09:55:38,156 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 2764d7f8155e1e5b0d641eebe063fa00 2023-04-09 09:55:38,156 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 2764d7f8155e1e5b0d641eebe063fa00 2023-04-09 09:55:38,157 INFO [StoreOpener-2764d7f8155e1e5b0d641eebe063fa00-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 2764d7f8155e1e5b0d641eebe063fa00 2023-04-09 09:55:38,158 DEBUG [StoreOpener-2764d7f8155e1e5b0d641eebe063fa00-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A 2023-04-09 09:55:38,158 DEBUG [StoreOpener-2764d7f8155e1e5b0d641eebe063fa00-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A 2023-04-09 09:55:38,158 INFO [StoreOpener-2764d7f8155e1e5b0d641eebe063fa00-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-09 09:55:38,159 INFO [StoreOpener-2764d7f8155e1e5b0d641eebe063fa00-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2764d7f8155e1e5b0d641eebe063fa00 columnFamilyName A 2023-04-09 09:55:38,159 INFO [StoreOpener-2764d7f8155e1e5b0d641eebe063fa00-1] regionserver.HStore(310): Store=2764d7f8155e1e5b0d641eebe063fa00/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-09 09:55:38,159 INFO [StoreOpener-2764d7f8155e1e5b0d641eebe063fa00-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 2764d7f8155e1e5b0d641eebe063fa00 2023-04-09 09:55:38,160 DEBUG [StoreOpener-2764d7f8155e1e5b0d641eebe063fa00-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B 2023-04-09 09:55:38,160 DEBUG [StoreOpener-2764d7f8155e1e5b0d641eebe063fa00-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B 2023-04-09 09:55:38,160 INFO [StoreOpener-2764d7f8155e1e5b0d641eebe063fa00-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-09 09:55:38,160 INFO [StoreOpener-2764d7f8155e1e5b0d641eebe063fa00-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2764d7f8155e1e5b0d641eebe063fa00 columnFamilyName B 2023-04-09 09:55:38,161 INFO [StoreOpener-2764d7f8155e1e5b0d641eebe063fa00-1] regionserver.HStore(310): Store=2764d7f8155e1e5b0d641eebe063fa00/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-09 09:55:38,161 INFO [StoreOpener-2764d7f8155e1e5b0d641eebe063fa00-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 2764d7f8155e1e5b0d641eebe063fa00 2023-04-09 09:55:38,162 DEBUG [StoreOpener-2764d7f8155e1e5b0d641eebe063fa00-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C 2023-04-09 09:55:38,162 DEBUG [StoreOpener-2764d7f8155e1e5b0d641eebe063fa00-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C 2023-04-09 09:55:38,162 INFO [StoreOpener-2764d7f8155e1e5b0d641eebe063fa00-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-09 09:55:38,162 INFO [StoreOpener-2764d7f8155e1e5b0d641eebe063fa00-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2764d7f8155e1e5b0d641eebe063fa00 columnFamilyName C 2023-04-09 09:55:38,162 INFO [StoreOpener-2764d7f8155e1e5b0d641eebe063fa00-1] regionserver.HStore(310): Store=2764d7f8155e1e5b0d641eebe063fa00/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-09 09:55:38,163 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:38,163 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00 2023-04-09 09:55:38,163 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00 2023-04-09 09:55:38,165 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-09 09:55:38,166 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 2764d7f8155e1e5b0d641eebe063fa00 2023-04-09 09:55:38,167 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-09 09:55:38,167 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 2764d7f8155e1e5b0d641eebe063fa00; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63451679, jitterRate=-0.05449630320072174}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-09 09:55:38,167 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 2764d7f8155e1e5b0d641eebe063fa00: 2023-04-09 09:55:38,168 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00., pid=49, masterSystemTime=1681034138153 2023-04-09 09:55:38,169 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:38,169 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:38,169 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=2764d7f8155e1e5b0d641eebe063fa00, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:38,170 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681034138169"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681034138169"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681034138169"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681034138169"}]},"ts":"1681034138169"} 2023-04-09 09:55:38,171 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=49, resume processing ppid=48 2023-04-09 09:55:38,171 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=49, ppid=48, state=SUCCESS; OpenRegionProcedure 2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 in 169 msec 2023-04-09 09:55:38,172 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=47 2023-04-09 09:55:38,172 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=47, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=2764d7f8155e1e5b0d641eebe063fa00, ASSIGN in 324 msec 2023-04-09 09:55:38,173 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-09 09:55:38,173 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681034138173"}]},"ts":"1681034138173"} 2023-04-09 09:55:38,174 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-09 09:55:38,175 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-09 09:55:38,176 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=47, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1540 sec 2023-04-09 09:55:39,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-09 09:55:39,128 INFO [Listener at localhost/44563] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 47 completed 2023-04-09 09:55:39,129 DEBUG [Listener at localhost/44563] zookeeper.ReadOnlyZKClient(139): Connect 0x2a987907 to 127.0.0.1:57816 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-09 09:55:39,134 DEBUG [Listener at localhost/44563] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2df63a4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-09 09:55:39,135 DEBUG [Listener at localhost/44563] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-09 09:55:39,136 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59296, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-09 09:55:39,137 DEBUG [Listener at localhost/44563] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-09 09:55:39,139 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52448, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-09 09:55:39,140 DEBUG [Listener at localhost/44563] zookeeper.ReadOnlyZKClient(139): Connect 0x37705401 to 127.0.0.1:57816 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-09 09:55:39,143 DEBUG [Listener at localhost/44563] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5ec943a9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-09 09:55:39,143 DEBUG [Listener at localhost/44563] zookeeper.ReadOnlyZKClient(139): Connect 0x15880ffb to 127.0.0.1:57816 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-09 09:55:39,147 DEBUG [Listener at localhost/44563] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@20ed9117, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-09 09:55:39,147 DEBUG [Listener at localhost/44563] zookeeper.ReadOnlyZKClient(139): Connect 0x052203cb to 127.0.0.1:57816 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-09 09:55:39,152 DEBUG [Listener at localhost/44563] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@50002343, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-09 09:55:39,152 DEBUG [Listener at localhost/44563] zookeeper.ReadOnlyZKClient(139): Connect 0x420f945a to 127.0.0.1:57816 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-09 09:55:39,155 DEBUG [Listener at localhost/44563] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5726fb8b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-09 09:55:39,155 DEBUG [Listener at localhost/44563] zookeeper.ReadOnlyZKClient(139): Connect 0x51b8df2a to 127.0.0.1:57816 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-09 09:55:39,158 DEBUG [Listener at localhost/44563] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6f2b3c03, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-09 09:55:39,159 DEBUG [Listener at localhost/44563] zookeeper.ReadOnlyZKClient(139): Connect 0x70e2be04 to 127.0.0.1:57816 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-09 09:55:39,162 DEBUG [Listener at localhost/44563] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@461f5f3f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-09 09:55:39,162 DEBUG [Listener at localhost/44563] zookeeper.ReadOnlyZKClient(139): Connect 0x067ea274 to 127.0.0.1:57816 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-09 09:55:39,166 DEBUG [Listener at localhost/44563] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@63093e21, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-09 09:55:39,166 DEBUG [Listener at localhost/44563] zookeeper.ReadOnlyZKClient(139): Connect 0x78592df0 to 127.0.0.1:57816 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-09 09:55:39,170 DEBUG [Listener at localhost/44563] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@b475d8d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-09 09:55:39,170 DEBUG [Listener at localhost/44563] zookeeper.ReadOnlyZKClient(139): Connect 0x659c68ab to 127.0.0.1:57816 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-09 09:55:39,174 DEBUG [Listener at localhost/44563] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@549bef45, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-09 09:55:39,174 DEBUG [Listener at localhost/44563] zookeeper.ReadOnlyZKClient(139): Connect 0x680e773e to 127.0.0.1:57816 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-09 09:55:39,177 DEBUG [Listener at localhost/44563] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6ed183a4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-09 09:55:39,184 DEBUG [hconnection-0x54040139-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-09 09:55:39,184 DEBUG [hconnection-0x539cbb5c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-09 09:55:39,184 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:39,185 DEBUG [hconnection-0x23bb825f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-09 09:55:39,185 DEBUG [hconnection-0x4cf1c817-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-09 09:55:39,185 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59316, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-09 09:55:39,185 DEBUG [hconnection-0x33d11841-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-09 09:55:39,186 DEBUG [hconnection-0x9c83745-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-09 09:55:39,186 DEBUG [hconnection-0x11f8fe3e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-09 09:55:39,186 DEBUG [hconnection-0x3af3e5da-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-09 09:55:39,188 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59342, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-09 09:55:39,188 DEBUG [hconnection-0x6e33227-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-09 09:55:39,191 DEBUG [hconnection-0x5a829bee-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-09 09:55:39,191 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59328, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-09 09:55:39,191 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59302, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-09 09:55:39,191 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59348, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-09 09:55:39,191 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59344, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-09 09:55:39,191 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59326, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-09 09:55:39,194 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59358, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-09 09:55:39,201 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59346, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-09 09:55:39,201 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59368, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-09 09:55:39,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:39,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:39,202 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:39,202 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:39,202 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:39,202 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:39,205 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:39,205 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:39,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 2764d7f8155e1e5b0d641eebe063fa00 2023-04-09 09:55:39,210 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2764d7f8155e1e5b0d641eebe063fa00 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-09 09:55:39,211 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:39,211 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:39,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:39,211 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:39,212 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:39,212 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:39,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:39,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:39,213 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:39,213 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=A 2023-04-09 09:55:39,213 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:39,213 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=B 2023-04-09 09:55:39,213 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:39,213 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=C 2023-04-09 09:55:39,213 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:39,215 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:39,215 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:39,215 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:39,215 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:39,215 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:39,215 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:39,215 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:39,215 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:39,215 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:39,215 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. as already flushing 2023-04-09 09:55:39,216 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:39,216 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:39,216 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:39,216 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,216 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:39,216 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:39,217 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,217 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:39,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:39,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:39,218 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:39,218 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:39,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:39,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:39,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:39,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:39,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:39,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48b85c95[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:39,220 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:39,221 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:39,224 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:39,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:39,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:39,224 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:39,224 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:39,224 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:39,225 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:39,226 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:39,226 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:39,228 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,228 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:39,228 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:39,228 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:39,228 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:39,228 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:39,228 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:39,228 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:39,231 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:39,231 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:39,231 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:39,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:39,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:39,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:39,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:39,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:39,231 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:39,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:39,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:39,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:39,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:39,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:39,232 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:39,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:39,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:39,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,235 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034139229, max timestamp=1681034139233Num uniques 859; 2023-04-09 09:55:39,235 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:55:39,235 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034139229, max timestamp=1681034139233Num uniques 859; 2023-04-09 09:55:39,235 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:55:39,235 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034139229, max timestamp=1681034139233Num uniques 859; 2023-04-09 09:55:39,236 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:55:39,236 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:39,236 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:39,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:39,236 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:39,236 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:39,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:39,236 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:39,236 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:39,237 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:39,237 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:39,237 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:39,237 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:39,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:39,237 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:39,237 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:39,237 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:39,237 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:39,239 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/5b7924b6347b4e4a9c093b9ca8268e27 2023-04-09 09:55:39,250 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:39,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59344 deadline: 1681034199241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,251 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:39,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59302 deadline: 1681034199242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,251 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:39,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59316 deadline: 1681034199247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,251 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:39,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59328 deadline: 1681034199248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,251 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:39,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59348 deadline: 1681034199249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,337 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:39,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:39,341 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:39,352 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:39,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59302 deadline: 1681034199352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,353 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:39,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59344 deadline: 1681034199352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,354 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:39,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59316 deadline: 1681034199352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,354 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:39,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59348 deadline: 1681034199352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,355 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:39,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59328 deadline: 1681034199353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:39,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:39,376 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:39,376 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:39,376 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:39,376 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:39,377 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:39,377 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:39,378 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:39,378 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:39,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:39,379 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,379 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:39,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:39,379 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:39,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:39,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:39,379 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:39,392 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:39,392 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:39,392 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:39,392 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:39,392 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:39,396 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:39,396 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:39,396 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:39,396 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:39,396 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. as already flushing 2023-04-09 09:55:39,396 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:39,396 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:39,396 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:39,396 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,403 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:39,403 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:39,405 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:39,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:39,405 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:39,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:39,405 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:39,406 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:39,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:39,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:39,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:39,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:39,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5943e24d[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:39,407 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:39,407 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:39,409 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:39,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:39,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:39,409 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:39,410 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:39,410 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:39,410 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,410 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:39,412 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,412 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:39,412 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:39,412 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:39,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:39,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:39,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:39,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:39,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:39,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:39,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:39,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:39,416 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:39,416 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:39,416 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:39,416 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:39,416 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:39,416 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:39,418 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:39,418 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:39,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:39,418 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:39,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:39,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:39,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:39,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:39,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:39,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:39,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:39,419 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:39,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:39,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:39,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:39,420 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:39,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:39,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,422 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:39,422 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:39,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:39,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:39,422 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,422 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:39,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:39,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:39,422 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:39,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:39,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:39,422 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:39,422 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:39,422 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:39,422 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:39,422 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,423 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:39,423 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:39,423 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:39,523 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:39,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:39,525 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:39,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:39,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:39,528 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:39,528 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:39,528 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:39,528 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:39,528 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:39,528 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:39,530 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:39,530 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:39,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:39,531 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:39,531 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:39,531 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:39,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:39,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:39,531 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:39,532 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:39,532 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:39,532 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:39,532 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:39,532 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:39,536 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:39,536 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:39,536 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:39,536 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:39,536 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. as already flushing 2023-04-09 09:55:39,536 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:39,536 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:39,536 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:39,536 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,537 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:39,537 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:39,539 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:39,539 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:39,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:39,539 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:39,539 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:39,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:39,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:39,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:39,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:39,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:39,540 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:39,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5712ed48[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:39,540 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:39,542 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:39,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:39,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:39,542 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:39,542 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:39,542 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:39,542 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,542 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:39,546 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:39,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:39,546 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:39,547 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:39,547 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:39,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:39,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:39,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:39,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:39,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:39,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:39,550 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:39,550 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:39,550 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:39,550 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:39,550 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:39,550 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:39,551 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:39,551 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:39,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:39,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:39,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:39,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:39,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:39,551 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:39,551 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:39,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:39,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:39,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:39,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:39,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:39,553 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:39,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:39,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:39,555 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,555 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:39,555 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:39,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:39,555 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:39,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:39,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:39,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:39,555 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:39,555 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,555 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:39,555 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:39,555 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:39,555 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:39,555 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:39,555 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:39,555 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:39,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:39,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:39,558 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:39,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59344 deadline: 1681034199555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,559 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:39,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59302 deadline: 1681034199555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,559 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:39,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59348 deadline: 1681034199556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,559 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:39,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59328 deadline: 1681034199556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,559 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:39,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59316 deadline: 1681034199556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,656 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:39,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:39,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:39,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:39,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:39,658 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:39,658 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:39,658 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:39,658 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:39,659 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:39,659 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:39,660 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:39,660 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:39,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:39,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:39,660 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,660 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:39,661 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:39,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:39,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:39,661 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:39,662 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:39,662 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:39,662 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:39,662 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:39,662 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:39,662 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:39,662 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:39,662 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:39,662 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:39,662 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. as already flushing 2023-04-09 09:55:39,662 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:39,662 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:39,662 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:39,663 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,663 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:39,663 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:39,665 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,665 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:39,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:39,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:39,665 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:39,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:39,665 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:39,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:39,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:39,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:39,666 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:39,666 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d38eeda[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:39,666 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:39,666 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:39,667 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:39,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:39,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:39,667 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:39,667 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:39,667 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,667 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:39,667 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:39,668 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/bd6937f13a3a42079cf442d73f1b3c20 2023-04-09 09:55:39,669 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,669 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:39,669 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:39,669 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:39,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:39,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:39,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:39,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:39,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:39,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:39,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:39,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:39,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:39,672 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:39,672 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:39,672 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:39,672 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:39,672 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:39,673 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:39,673 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:39,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:39,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:39,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:39,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:39,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:39,673 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:39,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:39,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:39,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:39,674 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:39,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:39,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:39,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:39,676 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:39,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:39,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,678 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:39,678 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:39,678 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,678 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:39,678 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:39,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:39,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:39,678 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:39,678 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,678 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:39,679 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:39,679 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:39,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:39,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:39,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:39,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:39,680 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:39,681 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:39,681 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:39,681 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/1dcb7efa637c4a20a749bb782e2d646f 2023-04-09 09:55:39,685 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/5b7924b6347b4e4a9c093b9ca8268e27 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/5b7924b6347b4e4a9c093b9ca8268e27 2023-04-09 09:55:39,688 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/5b7924b6347b4e4a9c093b9ca8268e27, entries=150, sequenceid=12, filesize=11.6 K 2023-04-09 09:55:39,689 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/bd6937f13a3a42079cf442d73f1b3c20 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/bd6937f13a3a42079cf442d73f1b3c20 2023-04-09 09:55:39,692 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/bd6937f13a3a42079cf442d73f1b3c20, entries=150, sequenceid=12, filesize=11.6 K 2023-04-09 09:55:39,693 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/1dcb7efa637c4a20a749bb782e2d646f as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/1dcb7efa637c4a20a749bb782e2d646f 2023-04-09 09:55:39,697 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/1dcb7efa637c4a20a749bb782e2d646f, entries=150, sequenceid=12, filesize=11.6 K 2023-04-09 09:55:39,697 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 2764d7f8155e1e5b0d641eebe063fa00 in 487ms, sequenceid=12, compaction requested=false 2023-04-09 09:55:39,698 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2764d7f8155e1e5b0d641eebe063fa00: 2023-04-09 09:55:39,781 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:39,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:39,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:39,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:39,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:39,784 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:39,784 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:39,784 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:39,784 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:39,785 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:39,785 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:39,786 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:39,786 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:39,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:39,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:39,786 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,787 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:39,787 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:39,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:39,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:39,787 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:39,787 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:39,787 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:39,787 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:39,788 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:39,788 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:39,788 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:39,788 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:39,788 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:39,788 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:39,788 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2764d7f8155e1e5b0d641eebe063fa00 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-09 09:55:39,788 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=A 2023-04-09 09:55:39,788 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:55:39,788 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=B 2023-04-09 09:55:39,788 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:55:39,788 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=C 2023-04-09 09:55:39,788 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:55:39,797 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/8eb41fcb814042ba8b70504eeb903c92 2023-04-09 09:55:39,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 2764d7f8155e1e5b0d641eebe063fa00 2023-04-09 09:55:39,862 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. as already flushing 2023-04-09 09:55:39,873 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:39,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59302 deadline: 1681034199868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,873 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:39,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59344 deadline: 1681034199869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,876 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:39,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59348 deadline: 1681034199870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,876 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:39,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59328 deadline: 1681034199871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,876 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:39,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59316 deadline: 1681034199871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,979 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:39,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59302 deadline: 1681034199974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,979 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:39,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59344 deadline: 1681034199974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,981 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:39,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59348 deadline: 1681034199977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,981 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:39,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59328 deadline: 1681034199977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:39,981 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:39,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59316 deadline: 1681034199977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:40,184 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:40,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59302 deadline: 1681034200180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:40,184 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:40,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59344 deadline: 1681034200180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:40,185 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:40,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59328 deadline: 1681034200182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:40,185 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:40,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59348 deadline: 1681034200182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:40,185 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:40,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59316 deadline: 1681034200183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:40,206 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/b8ce9346ce7545bba00c6d9d16b094e4 2023-04-09 09:55:40,489 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:40,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59302 deadline: 1681034200485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:40,489 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:40,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59348 deadline: 1681034200486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:40,490 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:40,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59316 deadline: 1681034200486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:40,491 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:40,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59328 deadline: 1681034200487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:40,491 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:40,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59344 deadline: 1681034200488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:40,616 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/1da5ee03214744829ed767d86420c4f3 2023-04-09 09:55:40,620 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/8eb41fcb814042ba8b70504eeb903c92 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/8eb41fcb814042ba8b70504eeb903c92 2023-04-09 09:55:40,624 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/8eb41fcb814042ba8b70504eeb903c92, entries=150, sequenceid=48, filesize=11.6 K 2023-04-09 09:55:40,625 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/b8ce9346ce7545bba00c6d9d16b094e4 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/b8ce9346ce7545bba00c6d9d16b094e4 2023-04-09 09:55:40,629 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/b8ce9346ce7545bba00c6d9d16b094e4, entries=150, sequenceid=48, filesize=11.6 K 2023-04-09 09:55:40,630 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/1da5ee03214744829ed767d86420c4f3 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/1da5ee03214744829ed767d86420c4f3 2023-04-09 09:55:40,633 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/1da5ee03214744829ed767d86420c4f3, entries=150, sequenceid=48, filesize=11.6 K 2023-04-09 09:55:40,633 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=60.38 KB/61830 for 2764d7f8155e1e5b0d641eebe063fa00 in 845ms, sequenceid=48, compaction requested=false 2023-04-09 09:55:40,633 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2764d7f8155e1e5b0d641eebe063fa00: 2023-04-09 09:55:40,633 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:40,633 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:40,633 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:40,633 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:40,634 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:40,634 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:40,635 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:40,635 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:40,635 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:40,635 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:40,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:40,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:40,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:40,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:40,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:40,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:40,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:40,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:40,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:40,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ee471c[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:40,637 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:40,637 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:40,639 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:40,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:40,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:40,639 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:40,639 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:40,639 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:40,639 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:40,639 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:40,640 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:40,640 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:40,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:40,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:40,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:40,641 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:40,641 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:40,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:40,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:40,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:40,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:40,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:40,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:40,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:40,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:40,643 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:40,643 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:40,643 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:40,643 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:40,643 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:40,643 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:40,645 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:40,645 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:40,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:40,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:40,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:40,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:40,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:40,646 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:40,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:40,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:40,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:40,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:40,647 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:40,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:40,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:40,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:40,648 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:40,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:40,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:40,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:40,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:40,650 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:40,650 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:40,650 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:40,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:40,650 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:40,650 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:40,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:40,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:40,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:40,650 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:40,650 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:40,650 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:40,650 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:40,650 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:40,650 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:40,650 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:40,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:40,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:40,750 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:40,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:40,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:40,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:40,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:40,754 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:40,754 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:40,754 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:40,754 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:40,755 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:40,755 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:40,756 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:40,756 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:40,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:40,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:40,756 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:40,756 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:40,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:40,756 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:40,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:40,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:40,757 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:40,757 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:40,757 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:40,757 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:40,757 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:40,757 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:40,760 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:40,760 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:40,760 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:40,760 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:40,760 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2764d7f8155e1e5b0d641eebe063fa00 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-09 09:55:40,760 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=A 2023-04-09 09:55:40,760 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:40,760 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=B 2023-04-09 09:55:40,760 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:40,760 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=C 2023-04-09 09:55:40,760 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:40,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 2764d7f8155e1e5b0d641eebe063fa00 2023-04-09 09:55:40,994 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. as already flushing 2023-04-09 09:55:41,016 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:41,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59316 deadline: 1681034201011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,016 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:41,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59348 deadline: 1681034201012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,016 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:41,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59328 deadline: 1681034201012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,019 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:41,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59344 deadline: 1681034201014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,020 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:41,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59302 deadline: 1681034201015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,119 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:41,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59316 deadline: 1681034201117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,119 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:41,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59348 deadline: 1681034201117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,119 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:41,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59328 deadline: 1681034201117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,124 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:41,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59344 deadline: 1681034201120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,125 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:41,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59302 deadline: 1681034201120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,175 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=60 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/8ba5a6c9869d4544ad3f5e08c68a6bb8 2023-04-09 09:55:41,264 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-09 09:55:41,325 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:41,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59316 deadline: 1681034201321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,325 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:41,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59348 deadline: 1681034201321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,326 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:41,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59328 deadline: 1681034201321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,328 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:41,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59344 deadline: 1681034201325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,331 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:41,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59302 deadline: 1681034201326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,585 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=60 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/47fecacfa69c4fc4a28aeff4cab0d908 2023-04-09 09:55:41,601 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=60 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/36e278ca57bf495ba3b86734b3401779 2023-04-09 09:55:41,606 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/8ba5a6c9869d4544ad3f5e08c68a6bb8 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/8ba5a6c9869d4544ad3f5e08c68a6bb8 2023-04-09 09:55:41,610 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/8ba5a6c9869d4544ad3f5e08c68a6bb8, entries=150, sequenceid=60, filesize=11.6 K 2023-04-09 09:55:41,611 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/47fecacfa69c4fc4a28aeff4cab0d908 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/47fecacfa69c4fc4a28aeff4cab0d908 2023-04-09 09:55:41,615 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/47fecacfa69c4fc4a28aeff4cab0d908, entries=150, sequenceid=60, filesize=11.6 K 2023-04-09 09:55:41,616 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/36e278ca57bf495ba3b86734b3401779 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/36e278ca57bf495ba3b86734b3401779 2023-04-09 09:55:41,621 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/36e278ca57bf495ba3b86734b3401779, entries=150, sequenceid=60, filesize=11.6 K 2023-04-09 09:55:41,622 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 2764d7f8155e1e5b0d641eebe063fa00 in 862ms, sequenceid=60, compaction requested=true 2023-04-09 09:55:41,622 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2764d7f8155e1e5b0d641eebe063fa00: 2023-04-09 09:55:41,622 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:41,622 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:41,622 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:41,622 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,622 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:41,622 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:41,623 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:41,623 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:41,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:41,624 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:41,624 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:41,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:41,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:41,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:41,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:41,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:41,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41f98c26[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:41,625 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:41,625 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:41,628 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:41,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:41,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:41,628 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:41,628 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:41,628 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:41,628 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,628 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:41,629 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,629 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:41,629 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:41,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:41,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:41,629 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:41,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:41,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:41,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:41,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:41,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:41,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:41,632 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:41,632 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:41,632 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:41,632 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034141010, max timestamp=1681034141630Num uniques 859; 2023-04-09 09:55:41,632 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:41,632 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:55:41,632 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:41,632 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:41,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 2764d7f8155e1e5b0d641eebe063fa00 2023-04-09 09:55:41,632 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2764d7f8155e1e5b0d641eebe063fa00 3/3 column families, dataSize=89.45 KB heapSize=262.11 KB 2023-04-09 09:55:41,632 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=A 2023-04-09 09:55:41,632 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:55:41,632 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=B 2023-04-09 09:55:41,632 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:55:41,632 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=C 2023-04-09 09:55:41,633 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:55:41,633 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:41,633 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:41,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:41,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:41,633 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:41,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:41,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:41,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:41,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:41,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:41,634 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:41,635 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:41,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:41,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:41,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:41,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,648 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:41,648 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:41,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:41,648 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:41,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:41,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:41,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:41,649 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:41,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:41,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:41,649 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:41,649 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:41,649 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,649 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:41,649 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:41,649 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:41,649 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:41,649 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:41,649 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:41,655 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:41,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59316 deadline: 1681034201651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,661 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:41,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59302 deadline: 1681034201651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,661 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:41,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59348 deadline: 1681034201651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,662 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:41,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59328 deadline: 1681034201655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,662 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:41,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59344 deadline: 1681034201655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,749 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:41,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:41,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:41,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:41,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:41,752 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:41,753 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:41,753 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:41,753 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:41,753 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:41,753 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:41,754 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,754 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:41,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:41,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:41,754 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,754 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:41,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:41,755 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:41,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:41,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:41,755 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:41,755 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:41,755 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:41,755 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:41,755 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:41,755 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:41,756 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:41,756 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:41,756 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:41,756 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:41,756 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. as already flushing 2023-04-09 09:55:41,756 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:41,756 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:41,756 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:41,756 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,756 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:41,756 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:41,762 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:41,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59316 deadline: 1681034201756, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,762 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,762 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:41,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:41,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:41,763 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:41,763 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:41,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:41,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:41,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:41,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:41,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:41,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c141fc[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:41,764 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:41,764 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:41,765 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:41,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:41,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:41,765 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,765 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:41,765 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:41,765 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:41,765 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:41,766 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,766 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:41,766 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:41,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:41,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:41,766 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:41,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:41,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:41,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:41,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:41,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:41,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:41,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:41,768 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:41,768 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:41,768 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:41,768 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:41,768 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:41,770 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:41,770 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:41,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:41,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:41,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:41,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:41,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:41,770 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:41,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:41,770 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:41,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:41,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:41,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:41,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:41,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:41,771 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:41,772 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:41,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:41,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59302 deadline: 1681034201762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,772 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:41,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59348 deadline: 1681034201762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,772 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:41,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59328 deadline: 1681034201763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,773 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:41,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59344 deadline: 1681034201763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,774 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,774 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:41,774 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:41,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:41,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:41,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:41,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:41,774 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:41,774 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,775 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:41,775 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:41,775 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:41,775 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:41,774 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:41,775 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:41,775 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:41,774 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:41,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:41,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:41,875 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:41,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:41,877 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:41,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:41,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:41,878 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:41,878 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:41,878 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:41,878 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:41,879 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:41,879 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:41,880 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,880 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:41,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:41,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:41,880 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,880 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:41,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:41,881 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:41,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:41,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:41,881 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:41,882 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:41,882 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:41,882 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:41,882 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:41,882 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:41,882 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:41,882 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:41,882 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:41,882 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:41,882 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. as already flushing 2023-04-09 09:55:41,882 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:41,882 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:41,882 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:41,882 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,882 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:41,882 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:41,883 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,883 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:41,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:41,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:41,884 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:41,884 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:41,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:41,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:41,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:41,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:41,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:41,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b452120[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:41,885 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:41,885 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:41,886 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:41,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:41,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:41,886 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:41,886 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:41,886 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:41,886 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,886 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:41,889 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:41,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:41,889 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:41,889 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:41,889 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:41,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:41,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:41,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:41,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:41,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:41,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:41,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:41,891 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:41,891 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:41,891 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:41,891 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:41,891 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:41,893 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:41,893 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:41,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:41,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:41,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:41,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:41,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:41,893 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:41,893 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:41,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:41,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:41,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:41,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:41,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:41,894 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:41,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:41,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:41,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,896 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,896 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:41,896 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:41,896 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:41,896 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:41,896 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,896 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:41,897 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:41,897 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:41,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:41,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:41,896 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:41,897 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:41,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:41,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:41,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:41,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:41,897 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:41,897 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:41,967 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:41,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59316 deadline: 1681034201963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,977 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:41,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59302 deadline: 1681034201973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,977 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:41,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59348 deadline: 1681034201973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,977 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:41,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59328 deadline: 1681034201973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,979 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:41,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59344 deadline: 1681034201974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:41,996 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-09 09:55:41,997 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:41,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:42,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:42,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:42,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:42,003 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:42,003 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:42,003 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:42,003 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:42,003 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:42,003 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:42,005 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,005 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:42,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:42,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:42,005 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,005 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:42,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:42,005 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:42,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:42,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:42,006 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:42,006 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:42,006 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:42,006 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:42,006 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:42,006 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:42,006 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:42,006 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:42,007 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:42,007 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:42,007 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. as already flushing 2023-04-09 09:55:42,007 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:42,007 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:42,007 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:42,007 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,008 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:42,008 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:42,009 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,009 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:42,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:42,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:42,009 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:42,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:42,010 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:42,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:42,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:42,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:42,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:42,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e6659f3[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:42,011 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:42,011 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:42,012 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:42,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:42,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:42,012 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:42,012 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:42,012 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:42,012 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,012 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:42,015 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:42,015 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:42,015 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,015 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:42,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:42,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:42,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:42,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:42,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:42,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:42,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:42,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:42,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:42,017 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:42,017 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:42,017 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:42,017 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:42,017 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:42,018 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:42,018 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:42,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:42,018 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:42,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:42,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:42,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:42,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:42,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:42,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:42,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:42,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:42,020 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:42,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:42,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:42,021 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:42,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:42,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,027 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,027 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:42,027 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:42,027 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,027 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:42,027 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:42,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:42,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:42,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:42,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:42,027 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:42,027 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:42,027 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:42,028 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:42,028 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:42,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:42,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:42,028 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:42,028 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:42,040 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=81 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/1481470a0253411aaeea374d1d287e89 2023-04-09 09:55:42,050 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=81 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/92e11b6c58d04fb9bcf063522a8efedc 2023-04-09 09:55:42,059 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=81 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/dee0848ce6324834b89e15700dd0703c 2023-04-09 09:55:42,062 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/1481470a0253411aaeea374d1d287e89 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/1481470a0253411aaeea374d1d287e89 2023-04-09 09:55:42,066 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/1481470a0253411aaeea374d1d287e89, entries=200, sequenceid=81, filesize=13.9 K 2023-04-09 09:55:42,066 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/92e11b6c58d04fb9bcf063522a8efedc as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/92e11b6c58d04fb9bcf063522a8efedc 2023-04-09 09:55:42,069 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/92e11b6c58d04fb9bcf063522a8efedc, entries=150, sequenceid=81, filesize=11.6 K 2023-04-09 09:55:42,070 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/dee0848ce6324834b89e15700dd0703c as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/dee0848ce6324834b89e15700dd0703c 2023-04-09 09:55:42,072 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/dee0848ce6324834b89e15700dd0703c, entries=150, sequenceid=81, filesize=11.6 K 2023-04-09 09:55:42,073 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~89.45 KB/91600, heapSize ~262.06 KB/268352, currentSize=87.22 KB/89310 for 2764d7f8155e1e5b0d641eebe063fa00 in 441ms, sequenceid=81, compaction requested=true 2023-04-09 09:55:42,073 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2764d7f8155e1e5b0d641eebe063fa00: 2023-04-09 09:55:42,073 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:55:42,073 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:55:42,073 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-09 09:55:42,073 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-09 09:55:42,073 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:55:42,074 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-09 09:55:42,074 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): 2764d7f8155e1e5b0d641eebe063fa00/B is initiating minor compaction (all files) 2023-04-09 09:55:42,074 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2764d7f8155e1e5b0d641eebe063fa00/B in TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:42,074 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/bd6937f13a3a42079cf442d73f1b3c20, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/b8ce9346ce7545bba00c6d9d16b094e4, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/47fecacfa69c4fc4a28aeff4cab0d908, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/92e11b6c58d04fb9bcf063522a8efedc] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp, totalSize=46.4 K 2023-04-09 09:55:42,084 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49884 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-09 09:55:42,084 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1912): 2764d7f8155e1e5b0d641eebe063fa00/A is initiating minor compaction (all files) 2023-04-09 09:55:42,084 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2764d7f8155e1e5b0d641eebe063fa00/A in TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:42,084 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/5b7924b6347b4e4a9c093b9ca8268e27, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/8eb41fcb814042ba8b70504eeb903c92, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/8ba5a6c9869d4544ad3f5e08c68a6bb8, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/1481470a0253411aaeea374d1d287e89] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp, totalSize=48.7 K 2023-04-09 09:55:42,084 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 5b7924b6347b4e4a9c093b9ca8268e27, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1681034139197 2023-04-09 09:55:42,084 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting bd6937f13a3a42079cf442d73f1b3c20, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1681034139197 2023-04-09 09:55:42,085 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 8eb41fcb814042ba8b70504eeb903c92, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1681034139245 2023-04-09 09:55:42,085 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting b8ce9346ce7545bba00c6d9d16b094e4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1681034139245 2023-04-09 09:55:42,085 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 8ba5a6c9869d4544ad3f5e08c68a6bb8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1681034139869 2023-04-09 09:55:42,086 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 47fecacfa69c4fc4a28aeff4cab0d908, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1681034139869 2023-04-09 09:55:42,086 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 1481470a0253411aaeea374d1d287e89, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=81, earliestPutTs=1681034141010 2023-04-09 09:55:42,086 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 92e11b6c58d04fb9bcf063522a8efedc, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=81, earliestPutTs=1681034141010 2023-04-09 09:55:42,094 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2764d7f8155e1e5b0d641eebe063fa00#A#compaction#429 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:55:42,097 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): 2764d7f8155e1e5b0d641eebe063fa00#B#compaction#430 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:55:42,128 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:42,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:42,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:42,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:42,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:42,132 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:42,132 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:42,132 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:42,132 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:42,136 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:42,136 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:42,138 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:42,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:42,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:42,139 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:42,139 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:42,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:42,140 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,140 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:42,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:42,140 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:42,141 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:42,141 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:42,141 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:42,141 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:42,141 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:42,141 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:42,141 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:42,141 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:42,141 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:42,141 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2764d7f8155e1e5b0d641eebe063fa00 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-09 09:55:42,142 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=A 2023-04-09 09:55:42,142 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:42,142 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=B 2023-04-09 09:55:42,142 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:42,142 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=C 2023-04-09 09:55:42,142 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:42,154 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=97 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/c69111087cb0438fbb9f69408854a40e 2023-04-09 09:55:42,171 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=97 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/3d98b5d1614249cd8e4b848e3c4df59d 2023-04-09 09:55:42,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 2764d7f8155e1e5b0d641eebe063fa00 2023-04-09 09:55:42,271 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. as already flushing 2023-04-09 09:55:42,295 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:42,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59328 deadline: 1681034202289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,296 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:42,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59344 deadline: 1681034202291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,298 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:42,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59302 deadline: 1681034202292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,300 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:42,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59348 deadline: 1681034202292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,300 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:42,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59316 deadline: 1681034202293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,401 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:42,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59328 deadline: 1681034202396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,401 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:42,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59344 deadline: 1681034202397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,403 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:42,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59302 deadline: 1681034202399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,404 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:42,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59348 deadline: 1681034202401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,404 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:42,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59316 deadline: 1681034202401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,511 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/6b49c1b493f24ec9b68e962c013f676f as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/6b49c1b493f24ec9b68e962c013f676f 2023-04-09 09:55:42,514 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/0aae36b8d7e44545be01d0db6bdcf4df as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/0aae36b8d7e44545be01d0db6bdcf4df 2023-04-09 09:55:42,516 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2764d7f8155e1e5b0d641eebe063fa00/A of 2764d7f8155e1e5b0d641eebe063fa00 into 6b49c1b493f24ec9b68e962c013f676f(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:55:42,516 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2764d7f8155e1e5b0d641eebe063fa00: 2023-04-09 09:55:42,516 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00., storeName=2764d7f8155e1e5b0d641eebe063fa00/A, priority=12, startTime=1681034142073; duration=0sec 2023-04-09 09:55:42,516 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:55:42,516 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-09 09:55:42,517 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-09 09:55:42,517 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1912): 2764d7f8155e1e5b0d641eebe063fa00/C is initiating minor compaction (all files) 2023-04-09 09:55:42,517 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2764d7f8155e1e5b0d641eebe063fa00/C in TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:42,517 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/1dcb7efa637c4a20a749bb782e2d646f, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/1da5ee03214744829ed767d86420c4f3, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/36e278ca57bf495ba3b86734b3401779, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/dee0848ce6324834b89e15700dd0703c] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp, totalSize=46.4 K 2023-04-09 09:55:42,518 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 1dcb7efa637c4a20a749bb782e2d646f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1681034139197 2023-04-09 09:55:42,518 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 1da5ee03214744829ed767d86420c4f3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1681034139245 2023-04-09 09:55:42,518 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 36e278ca57bf495ba3b86734b3401779, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1681034139869 2023-04-09 09:55:42,519 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2764d7f8155e1e5b0d641eebe063fa00/B of 2764d7f8155e1e5b0d641eebe063fa00 into 0aae36b8d7e44545be01d0db6bdcf4df(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:55:42,519 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2764d7f8155e1e5b0d641eebe063fa00: 2023-04-09 09:55:42,519 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00., storeName=2764d7f8155e1e5b0d641eebe063fa00/B, priority=12, startTime=1681034142073; duration=0sec 2023-04-09 09:55:42,519 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting dee0848ce6324834b89e15700dd0703c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=81, earliestPutTs=1681034141010 2023-04-09 09:55:42,519 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:55:42,535 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2764d7f8155e1e5b0d641eebe063fa00#C#compaction#434 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:55:42,582 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=97 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/d1b03bebd97c4b11876568049a0cfed6 2023-04-09 09:55:42,585 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/c69111087cb0438fbb9f69408854a40e as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/c69111087cb0438fbb9f69408854a40e 2023-04-09 09:55:42,589 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/c69111087cb0438fbb9f69408854a40e, entries=150, sequenceid=97, filesize=11.6 K 2023-04-09 09:55:42,590 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/3d98b5d1614249cd8e4b848e3c4df59d as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/3d98b5d1614249cd8e4b848e3c4df59d 2023-04-09 09:55:42,593 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/3d98b5d1614249cd8e4b848e3c4df59d, entries=150, sequenceid=97, filesize=11.6 K 2023-04-09 09:55:42,593 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/d1b03bebd97c4b11876568049a0cfed6 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/d1b03bebd97c4b11876568049a0cfed6 2023-04-09 09:55:42,596 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/d1b03bebd97c4b11876568049a0cfed6, entries=150, sequenceid=97, filesize=11.6 K 2023-04-09 09:55:42,597 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 2764d7f8155e1e5b0d641eebe063fa00 in 456ms, sequenceid=97, compaction requested=false 2023-04-09 09:55:42,597 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2764d7f8155e1e5b0d641eebe063fa00: 2023-04-09 09:55:42,597 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:42,597 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:42,597 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:42,597 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,597 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:42,597 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:42,599 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,599 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:42,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:42,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:42,599 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:42,599 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:42,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:42,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:42,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:42,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:42,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:42,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33f36aa5[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:42,601 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:42,601 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:42,603 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:42,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:42,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:42,603 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:42,603 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:42,603 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:42,604 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,604 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:42,605 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:42,605 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:42,605 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,605 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:42,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:42,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:42,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:42,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:42,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:42,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:42,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:42,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 2764d7f8155e1e5b0d641eebe063fa00 2023-04-09 09:55:42,607 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2764d7f8155e1e5b0d641eebe063fa00 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-09 09:55:42,607 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=A 2023-04-09 09:55:42,607 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:42,607 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=B 2023-04-09 09:55:42,607 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:42,607 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=C 2023-04-09 09:55:42,607 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:42,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:42,608 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:42,608 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:42,608 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:42,608 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:42,608 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:42,608 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:42,614 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:42,614 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:42,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:42,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:42,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:42,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:42,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:42,614 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:42,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:42,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:42,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:42,615 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:42,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:42,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:42,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:42,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,616 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:42,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:42,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,618 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,618 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:42,618 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:42,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:42,618 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:42,618 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:42,618 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,618 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:42,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:42,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:42,618 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:42,618 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:42,619 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:42,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:42,619 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:42,619 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:42,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:42,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:42,627 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:42,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59302 deadline: 1681034202620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,628 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:42,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59348 deadline: 1681034202621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,628 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:42,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59344 deadline: 1681034202622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,628 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:42,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59328 deadline: 1681034202622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,632 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:42,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59316 deadline: 1681034202627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,719 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:42,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:42,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:42,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:42,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:42,722 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:42,722 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:42,722 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:42,722 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:42,723 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:42,723 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:42,725 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:42,725 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:42,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:42,725 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,725 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:42,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:42,725 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:42,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:42,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:42,726 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:42,726 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:42,726 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:42,726 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:42,726 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:42,726 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:42,727 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:42,727 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:42,727 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:42,727 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:42,727 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. as already flushing 2023-04-09 09:55:42,727 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:42,727 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:42,727 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:42,727 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,727 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:42,727 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:42,728 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:42,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:42,729 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:42,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:42,729 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:42,729 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:42,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:42,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:42,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:42,730 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:42,730 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38b61386[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:42,730 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:42,730 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:42,731 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:42,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:42,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:42,731 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:42,731 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,731 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:42,731 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:42,731 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:42,733 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,733 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:42,733 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:42,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:42,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:42,733 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:42,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:42,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:42,734 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:42,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:42,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59302 deadline: 1681034202728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,734 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:42,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59348 deadline: 1681034202729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,734 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:42,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:42,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59344 deadline: 1681034202729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,734 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:42,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59328 deadline: 1681034202729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:42,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:42,735 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:42,735 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:42,735 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:42,735 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:42,735 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:42,735 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:42,736 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:42,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59316 deadline: 1681034202733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,738 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:42,738 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:42,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:42,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:42,738 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:42,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:42,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:42,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:42,738 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:42,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:42,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:42,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:42,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:42,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:42,740 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:42,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:42,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:42,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,742 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,742 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:42,742 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:42,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:42,742 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:42,742 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:42,742 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:42,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:42,742 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:42,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:42,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:42,742 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:42,742 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,742 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:42,742 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:42,742 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:42,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:42,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:42,742 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:42,842 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:42,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:42,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:42,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:42,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:42,846 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:42,846 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:42,846 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:42,846 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:42,846 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:42,846 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:42,847 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:42,847 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:42,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:42,848 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,848 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:42,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:42,848 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:42,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:42,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:42,848 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:42,849 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:42,849 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:42,849 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:42,849 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:42,849 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:42,849 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:42,849 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:42,849 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:42,849 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:42,849 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. as already flushing 2023-04-09 09:55:42,849 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:42,849 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:42,849 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:42,849 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,849 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:42,849 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:42,850 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,850 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:42,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:42,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:42,851 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:42,851 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:42,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:42,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:42,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:42,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:42,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:42,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a987991[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:42,852 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:42,852 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:42,855 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:42,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:42,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:42,855 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:42,855 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:42,855 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,855 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:42,855 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:42,856 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,856 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:42,856 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:42,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:42,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:42,856 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:42,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:42,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:42,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:42,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:42,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:42,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:42,859 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:42,859 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:42,859 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:42,859 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:42,859 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:42,859 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:42,860 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:42,860 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:42,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:42,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:42,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:42,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:42,860 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:42,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:42,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:42,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:42,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:42,861 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:42,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:42,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:42,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:42,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,862 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:42,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:42,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,864 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,864 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:42,865 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:42,865 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:42,865 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:42,865 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:42,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:42,865 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:42,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:42,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:42,865 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:42,865 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:42,865 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:42,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:42,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:42,865 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:42,865 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:42,865 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:42,940 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:42,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59348 deadline: 1681034202935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,940 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:42,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59328 deadline: 1681034202935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,940 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:42,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59302 deadline: 1681034202936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,941 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:42,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59344 deadline: 1681034202936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,941 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:42,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59316 deadline: 1681034202937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,948 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/4b274876648540ff91277c8cf90982c5 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/4b274876648540ff91277c8cf90982c5 2023-04-09 09:55:42,951 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2764d7f8155e1e5b0d641eebe063fa00/C of 2764d7f8155e1e5b0d641eebe063fa00 into 4b274876648540ff91277c8cf90982c5(size=11.7 K), total size for store is 23.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:55:42,951 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2764d7f8155e1e5b0d641eebe063fa00: 2023-04-09 09:55:42,951 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00., storeName=2764d7f8155e1e5b0d641eebe063fa00/C, priority=12, startTime=1681034142073; duration=0sec 2023-04-09 09:55:42,951 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:55:42,965 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:42,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:42,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:42,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:42,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:42,969 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:42,969 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:42,969 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:42,969 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:42,969 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:42,969 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:42,974 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,974 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:42,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:42,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:42,974 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,974 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:42,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:42,975 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:42,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:42,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:42,975 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:42,975 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:42,975 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:42,975 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:42,975 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:42,976 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:42,976 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:42,976 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:42,976 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:42,976 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:42,976 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. as already flushing 2023-04-09 09:55:42,976 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:42,976 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:42,976 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:42,976 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,976 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:42,976 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:42,977 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:42,977 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:42,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:42,978 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:42,978 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:42,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:42,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:42,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:42,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:42,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:42,979 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:42,979 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:42,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56bd1af0[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:42,980 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:42,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:42,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:42,981 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:42,981 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:42,981 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:42,981 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,981 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:42,982 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,982 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:42,982 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:42,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:42,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:42,982 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:42,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:42,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:42,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:42,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:42,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:42,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:42,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:42,985 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:42,985 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:42,985 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:42,985 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:42,985 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:42,989 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:42,989 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:42,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:42,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:42,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:42,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:42,989 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:42,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:42,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:42,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:42,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:42,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:42,990 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:42,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:42,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:42,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,991 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:42,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:42,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,993 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,993 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:42,993 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:42,993 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:42,993 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:42,993 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:42,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:42,993 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:42,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:42,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:42,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:42,993 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:42,993 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:42,993 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:42,994 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:42,994 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:42,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:42,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:43,015 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=116 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/7981f64e45b44940a3df8fa4ccc2c45d 2023-04-09 09:55:43,024 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=116 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/d68b211003574a2d909f62b401f8398e 2023-04-09 09:55:43,034 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=116 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/549df63dab2140d1b35d9d2a7fa96b0c 2023-04-09 09:55:43,038 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/7981f64e45b44940a3df8fa4ccc2c45d as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/7981f64e45b44940a3df8fa4ccc2c45d 2023-04-09 09:55:43,041 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/7981f64e45b44940a3df8fa4ccc2c45d, entries=200, sequenceid=116, filesize=13.9 K 2023-04-09 09:55:43,042 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/d68b211003574a2d909f62b401f8398e as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/d68b211003574a2d909f62b401f8398e 2023-04-09 09:55:43,045 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/d68b211003574a2d909f62b401f8398e, entries=150, sequenceid=116, filesize=11.6 K 2023-04-09 09:55:43,045 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/549df63dab2140d1b35d9d2a7fa96b0c as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/549df63dab2140d1b35d9d2a7fa96b0c 2023-04-09 09:55:43,060 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/549df63dab2140d1b35d9d2a7fa96b0c, entries=150, sequenceid=116, filesize=11.6 K 2023-04-09 09:55:43,061 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 2764d7f8155e1e5b0d641eebe063fa00 in 454ms, sequenceid=116, compaction requested=true 2023-04-09 09:55:43,061 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2764d7f8155e1e5b0d641eebe063fa00: 2023-04-09 09:55:43,061 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-09 09:55:43,061 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:55:43,061 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:55:43,061 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:55:43,061 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-09 09:55:43,062 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35796 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-09 09:55:43,062 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): 2764d7f8155e1e5b0d641eebe063fa00/B is initiating minor compaction (all files) 2023-04-09 09:55:43,062 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38136 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-09 09:55:43,062 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2764d7f8155e1e5b0d641eebe063fa00/B in TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:43,063 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1912): 2764d7f8155e1e5b0d641eebe063fa00/A is initiating minor compaction (all files) 2023-04-09 09:55:43,063 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/0aae36b8d7e44545be01d0db6bdcf4df, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/3d98b5d1614249cd8e4b848e3c4df59d, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/d68b211003574a2d909f62b401f8398e] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp, totalSize=35.0 K 2023-04-09 09:55:43,063 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2764d7f8155e1e5b0d641eebe063fa00/A in TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:43,063 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/6b49c1b493f24ec9b68e962c013f676f, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/c69111087cb0438fbb9f69408854a40e, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/7981f64e45b44940a3df8fa4ccc2c45d] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp, totalSize=37.2 K 2023-04-09 09:55:43,063 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 0aae36b8d7e44545be01d0db6bdcf4df, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=81, earliestPutTs=1681034141010 2023-04-09 09:55:43,063 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 6b49c1b493f24ec9b68e962c013f676f, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=81, earliestPutTs=1681034141010 2023-04-09 09:55:43,063 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 3d98b5d1614249cd8e4b848e3c4df59d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1681034141641 2023-04-09 09:55:43,064 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting c69111087cb0438fbb9f69408854a40e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1681034141641 2023-04-09 09:55:43,064 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting d68b211003574a2d909f62b401f8398e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1681034142291 2023-04-09 09:55:43,064 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 7981f64e45b44940a3df8fa4ccc2c45d, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1681034142291 2023-04-09 09:55:43,075 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2764d7f8155e1e5b0d641eebe063fa00#A#compaction#438 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:55:43,077 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): 2764d7f8155e1e5b0d641eebe063fa00#B#compaction#439 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:55:43,094 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:43,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:43,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:43,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:43,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:43,108 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:43,108 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:43,108 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:43,108 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:43,109 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:43,109 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:43,110 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,110 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:43,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:43,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:43,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:43,111 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,111 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:43,111 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:43,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:43,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:43,112 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:43,112 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:43,112 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:43,112 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:43,112 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:43,112 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:43,113 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:43,113 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:43,113 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:43,113 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2764d7f8155e1e5b0d641eebe063fa00 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-09 09:55:43,113 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=A 2023-04-09 09:55:43,113 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:43,114 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=B 2023-04-09 09:55:43,114 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:43,114 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=C 2023-04-09 09:55:43,114 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:43,114 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/7e9d29ce8d6d4169b93888ade90afb0a as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/7e9d29ce8d6d4169b93888ade90afb0a 2023-04-09 09:55:43,123 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2764d7f8155e1e5b0d641eebe063fa00/B of 2764d7f8155e1e5b0d641eebe063fa00 into 7e9d29ce8d6d4169b93888ade90afb0a(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:55:43,123 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2764d7f8155e1e5b0d641eebe063fa00: 2023-04-09 09:55:43,123 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00., storeName=2764d7f8155e1e5b0d641eebe063fa00/B, priority=13, startTime=1681034143061; duration=0sec 2023-04-09 09:55:43,124 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:55:43,124 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-09 09:55:43,125 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35796 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-09 09:55:43,125 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): 2764d7f8155e1e5b0d641eebe063fa00/C is initiating minor compaction (all files) 2023-04-09 09:55:43,125 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2764d7f8155e1e5b0d641eebe063fa00/C in TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:43,125 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/4b274876648540ff91277c8cf90982c5, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/d1b03bebd97c4b11876568049a0cfed6, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/549df63dab2140d1b35d9d2a7fa96b0c] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp, totalSize=35.0 K 2023-04-09 09:55:43,125 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 4b274876648540ff91277c8cf90982c5, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=81, earliestPutTs=1681034141010 2023-04-09 09:55:43,125 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting d1b03bebd97c4b11876568049a0cfed6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1681034141641 2023-04-09 09:55:43,126 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 549df63dab2140d1b35d9d2a7fa96b0c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1681034142291 2023-04-09 09:55:43,141 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): 2764d7f8155e1e5b0d641eebe063fa00#C#compaction#441 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:55:43,142 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=132 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/0d9ea862f2164a0e8d2915e93f3cd501 2023-04-09 09:55:43,157 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/da297746c817402fbc14e6e570d0e6ca as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/da297746c817402fbc14e6e570d0e6ca 2023-04-09 09:55:43,160 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2764d7f8155e1e5b0d641eebe063fa00/C of 2764d7f8155e1e5b0d641eebe063fa00 into da297746c817402fbc14e6e570d0e6ca(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:55:43,160 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2764d7f8155e1e5b0d641eebe063fa00: 2023-04-09 09:55:43,161 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00., storeName=2764d7f8155e1e5b0d641eebe063fa00/C, priority=13, startTime=1681034143061; duration=0sec 2023-04-09 09:55:43,161 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:55:43,164 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=132 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/b0c76351160c4d7fa431fe929e7edfb4 2023-04-09 09:55:43,174 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=132 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/d8db39d6e8e742f4bb39934ff3fd1d87 2023-04-09 09:55:43,178 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/0d9ea862f2164a0e8d2915e93f3cd501 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/0d9ea862f2164a0e8d2915e93f3cd501 2023-04-09 09:55:43,183 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/0d9ea862f2164a0e8d2915e93f3cd501, entries=150, sequenceid=132, filesize=11.7 K 2023-04-09 09:55:43,184 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/b0c76351160c4d7fa431fe929e7edfb4 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/b0c76351160c4d7fa431fe929e7edfb4 2023-04-09 09:55:43,188 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/b0c76351160c4d7fa431fe929e7edfb4, entries=150, sequenceid=132, filesize=11.7 K 2023-04-09 09:55:43,189 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/d8db39d6e8e742f4bb39934ff3fd1d87 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/d8db39d6e8e742f4bb39934ff3fd1d87 2023-04-09 09:55:43,191 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/d8db39d6e8e742f4bb39934ff3fd1d87, entries=150, sequenceid=132, filesize=11.7 K 2023-04-09 09:55:43,192 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=0 B/0 for 2764d7f8155e1e5b0d641eebe063fa00 in 79ms, sequenceid=132, compaction requested=false 2023-04-09 09:55:43,192 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2764d7f8155e1e5b0d641eebe063fa00: 2023-04-09 09:55:43,192 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:43,192 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:43,192 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:43,192 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,192 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:43,193 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:43,195 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,195 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:43,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:43,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:43,195 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:43,196 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:43,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:43,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:43,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:43,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:43,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:43,201 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:43,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e33e4c0[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:43,201 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:43,202 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:43,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:43,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:43,202 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:43,202 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:43,202 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:43,202 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,202 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:43,203 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,203 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:43,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:43,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:43,203 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:43,203 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:43,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:43,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:43,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:43,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:43,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:43,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:43,206 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:43,206 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:43,206 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:43,206 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:43,206 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:43,206 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:43,208 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:43,208 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:43,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:43,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:43,208 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:43,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:43,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:43,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:43,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:43,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:43,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:43,209 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:43,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:43,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:43,210 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:43,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:43,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:43,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,213 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:43,213 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:43,213 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:43,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:43,214 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:43,214 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:43,214 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,214 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:43,214 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:43,214 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:43,214 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:43,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:43,214 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:43,214 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:43,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:43,214 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:43,214 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:43,214 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:43,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 2764d7f8155e1e5b0d641eebe063fa00 2023-04-09 09:55:43,249 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2764d7f8155e1e5b0d641eebe063fa00 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-09 09:55:43,250 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=A 2023-04-09 09:55:43,250 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:43,250 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=B 2023-04-09 09:55:43,250 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:43,250 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=C 2023-04-09 09:55:43,250 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:43,257 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=145 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/7e3567833c804b3db3968083cab07f45 2023-04-09 09:55:43,278 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:43,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59348 deadline: 1681034203269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,278 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681034143266, max timestamp=1681034143277Num uniques 859; 2023-04-09 09:55:43,278 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-09 09:55:43,279 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681034143266, max timestamp=1681034143277Num uniques 859; 2023-04-09 09:55:43,279 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681034143266, max timestamp=1681034143277Num uniques 859; 2023-04-09 09:55:43,279 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-09 09:55:43,279 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-09 09:55:43,301 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:43,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59302 deadline: 1681034203294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,303 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:43,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59344 deadline: 1681034203297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,303 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:43,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59316 deadline: 1681034203300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,307 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:43,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59328 deadline: 1681034203300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,314 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:43,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:43,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:43,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:43,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:43,318 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:43,318 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:43,318 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:43,318 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:43,318 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:43,318 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:43,319 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:43,319 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:43,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:43,320 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,320 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:43,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:43,320 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:43,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:43,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:43,321 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:43,321 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:43,321 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:43,321 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:43,321 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:43,321 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:43,321 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:43,321 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:43,321 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:43,321 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:43,321 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. as already flushing 2023-04-09 09:55:43,321 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:43,321 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:43,321 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:43,321 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,321 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:43,322 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:43,323 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,323 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:43,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:43,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:43,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:43,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:43,323 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:43,324 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:43,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:43,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:43,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:43,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7691f2a3[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:43,324 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:43,325 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:43,327 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:43,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:43,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:43,327 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:43,327 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:43,327 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:43,327 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,327 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:43,328 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,328 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:43,328 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:43,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:43,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:43,328 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:43,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:43,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:43,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:43,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:43,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:43,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:43,330 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:43,330 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:43,331 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:43,331 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:43,331 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:43,331 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:43,332 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:43,332 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:43,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:43,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:43,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:43,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:43,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:43,332 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:43,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:43,333 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:43,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:43,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:43,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:43,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:43,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:43,334 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:43,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:43,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,336 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:43,336 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,336 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:43,336 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:43,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:43,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:43,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:43,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:43,336 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:43,336 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:43,336 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,336 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:43,336 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:43,336 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:43,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:43,336 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:43,336 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:43,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:43,384 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:43,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59348 deadline: 1681034203379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,404 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:43,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59302 deadline: 1681034203402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,408 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:43,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59344 deadline: 1681034203404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,409 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:43,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59316 deadline: 1681034203404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,411 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:43,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59328 deadline: 1681034203408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,437 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:43,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:43,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:43,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:43,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:43,440 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:43,440 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:43,440 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:43,440 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:43,440 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:43,440 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:43,442 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:43,442 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:43,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:43,443 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,443 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:43,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:43,443 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:43,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:43,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:43,443 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:43,443 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:43,443 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:43,444 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:43,444 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:43,444 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:43,444 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:43,444 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:43,444 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:43,444 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:43,444 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. as already flushing 2023-04-09 09:55:43,444 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:43,444 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:43,444 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:43,444 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,444 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:43,445 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:43,448 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,448 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:43,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:43,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:43,449 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:43,449 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:43,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:43,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:43,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:43,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:43,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:43,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f5c5911[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:43,450 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:43,450 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:43,451 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:43,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:43,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:43,451 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:43,451 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:43,451 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:43,452 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,452 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:43,454 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:43,454 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:43,454 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,454 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:43,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:43,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:43,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:43,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:43,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:43,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:43,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:43,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:43,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:43,457 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:43,457 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:43,457 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:43,457 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:43,457 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:43,458 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:43,458 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:43,458 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:43,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:43,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:43,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:43,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:43,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:43,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:43,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:43,459 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:43,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:43,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:43,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:43,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:43,460 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:43,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:43,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,462 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,462 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:43,462 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:43,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:43,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:43,462 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:43,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:43,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:43,462 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:43,462 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:43,462 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:43,462 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:43,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:43,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:43,462 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:43,462 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,462 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:43,462 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:43,462 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:43,493 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/c9bd2efe5d5c4593925f772659d31c29 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/c9bd2efe5d5c4593925f772659d31c29 2023-04-09 09:55:43,497 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2764d7f8155e1e5b0d641eebe063fa00/A of 2764d7f8155e1e5b0d641eebe063fa00 into c9bd2efe5d5c4593925f772659d31c29(size=11.8 K), total size for store is 23.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:55:43,497 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2764d7f8155e1e5b0d641eebe063fa00: 2023-04-09 09:55:43,497 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00., storeName=2764d7f8155e1e5b0d641eebe063fa00/A, priority=13, startTime=1681034143061; duration=0sec 2023-04-09 09:55:43,498 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:55:43,562 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:43,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:43,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:43,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:43,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:43,565 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:43,565 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:43,565 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:43,565 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:43,566 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:43,566 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:43,567 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:43,567 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:43,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:43,567 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,567 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:43,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:43,568 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:43,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:43,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:43,568 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:43,568 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:43,569 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:43,569 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:43,569 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:43,569 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:43,569 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:43,569 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:43,569 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:43,569 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:43,569 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. as already flushing 2023-04-09 09:55:43,569 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:43,569 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:43,569 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:43,569 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,569 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:43,569 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:43,571 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,571 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:43,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:43,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:43,572 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:43,572 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:43,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:43,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:43,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:43,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:43,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:43,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5108b490[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:43,573 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:43,573 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:43,574 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:43,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:43,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:43,574 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:43,574 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,574 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:43,574 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:43,574 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:43,576 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:43,576 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,576 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:43,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:43,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:43,576 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:43,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:43,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:43,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:43,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:43,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:43,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:43,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:43,578 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:43,578 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:43,578 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:43,578 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:43,578 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:43,579 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:43,579 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:43,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:43,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:43,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:43,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:43,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:43,579 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:43,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:43,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:43,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:43,580 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:43,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:43,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:43,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:43,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,581 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:43,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:43,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,584 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,584 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:43,584 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:43,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:43,584 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:43,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:43,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:43,584 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:43,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:43,584 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:43,584 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:43,585 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:43,585 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:43,585 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,585 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:43,585 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:43,585 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:43,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:43,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:43,590 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:43,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59348 deadline: 1681034203585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,609 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:43,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59302 deadline: 1681034203605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,615 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:43,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59344 deadline: 1681034203610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,616 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:43,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59316 deadline: 1681034203611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,616 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:43,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59328 deadline: 1681034203611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,665 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=145 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/f20c9961caf6478e8485884e43e77040 2023-04-09 09:55:43,674 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=145 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/1f67a71e84ee44f9b2c3ef1a6326a073 2023-04-09 09:55:43,678 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/7e3567833c804b3db3968083cab07f45 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/7e3567833c804b3db3968083cab07f45 2023-04-09 09:55:43,681 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/7e3567833c804b3db3968083cab07f45, entries=200, sequenceid=145, filesize=14.1 K 2023-04-09 09:55:43,682 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/f20c9961caf6478e8485884e43e77040 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/f20c9961caf6478e8485884e43e77040 2023-04-09 09:55:43,684 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/f20c9961caf6478e8485884e43e77040, entries=150, sequenceid=145, filesize=11.8 K 2023-04-09 09:55:43,685 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:43,685 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/1f67a71e84ee44f9b2c3ef1a6326a073 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/1f67a71e84ee44f9b2c3ef1a6326a073 2023-04-09 09:55:43,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:43,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:43,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:43,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:43,688 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:43,688 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:43,688 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:43,688 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:43,688 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/1f67a71e84ee44f9b2c3ef1a6326a073, entries=150, sequenceid=145, filesize=11.8 K 2023-04-09 09:55:43,689 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:43,689 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:43,689 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 2764d7f8155e1e5b0d641eebe063fa00 in 440ms, sequenceid=145, compaction requested=true 2023-04-09 09:55:43,689 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2764d7f8155e1e5b0d641eebe063fa00: 2023-04-09 09:55:43,689 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:55:43,689 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:55:43,689 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-09 09:55:43,689 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-09 09:55:43,689 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:55:43,690 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,690 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:43,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:43,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:43,690 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,690 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:43,690 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36148 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-09 09:55:43,690 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38538 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-09 09:55:43,690 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): 2764d7f8155e1e5b0d641eebe063fa00/B is initiating minor compaction (all files) 2023-04-09 09:55:43,690 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1912): 2764d7f8155e1e5b0d641eebe063fa00/A is initiating minor compaction (all files) 2023-04-09 09:55:43,691 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2764d7f8155e1e5b0d641eebe063fa00/B in TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:43,691 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2764d7f8155e1e5b0d641eebe063fa00/A in TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:43,691 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/7e9d29ce8d6d4169b93888ade90afb0a, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/b0c76351160c4d7fa431fe929e7edfb4, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/f20c9961caf6478e8485884e43e77040] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp, totalSize=35.3 K 2023-04-09 09:55:43,691 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/c9bd2efe5d5c4593925f772659d31c29, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/0d9ea862f2164a0e8d2915e93f3cd501, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/7e3567833c804b3db3968083cab07f45] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp, totalSize=37.6 K 2023-04-09 09:55:43,691 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 7e9d29ce8d6d4169b93888ade90afb0a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1681034142291 2023-04-09 09:55:43,691 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting c9bd2efe5d5c4593925f772659d31c29, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1681034142291 2023-04-09 09:55:43,691 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting b0c76351160c4d7fa431fe929e7edfb4, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=132, earliestPutTs=1681034142619 2023-04-09 09:55:43,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:43,692 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 0d9ea862f2164a0e8d2915e93f3cd501, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=132, earliestPutTs=1681034142619 2023-04-09 09:55:43,692 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:43,692 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting f20c9961caf6478e8485884e43e77040, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=145, earliestPutTs=1681034143244 2023-04-09 09:55:43,692 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 7e3567833c804b3db3968083cab07f45, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=145, earliestPutTs=1681034143244 2023-04-09 09:55:43,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:43,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:43,692 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:43,692 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:43,692 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:43,693 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:43,693 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:43,693 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:43,693 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:43,693 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:43,693 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:43,693 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:43,693 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2764d7f8155e1e5b0d641eebe063fa00 3/3 column families, dataSize=134.18 KB heapSize=370.92 KB 2023-04-09 09:55:43,693 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=A 2023-04-09 09:55:43,693 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:55:43,693 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=B 2023-04-09 09:55:43,693 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:55:43,693 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=C 2023-04-09 09:55:43,693 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:55:43,697 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): 2764d7f8155e1e5b0d641eebe063fa00#B#compaction#448 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:55:43,698 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2764d7f8155e1e5b0d641eebe063fa00#A#compaction#449 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:55:43,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 2764d7f8155e1e5b0d641eebe063fa00 2023-04-09 09:55:43,897 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. as already flushing 2023-04-09 09:55:43,928 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:43,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59302 deadline: 1681034203917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,928 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:43,928 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:43,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59348 deadline: 1681034203918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59328 deadline: 1681034203918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,934 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:43,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59344 deadline: 1681034203926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:43,934 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:43,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59316 deadline: 1681034203927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,033 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:44,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59302 deadline: 1681034204029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,033 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:44,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59348 deadline: 1681034204029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,042 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:44,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59316 deadline: 1681034204037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,042 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:44,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59344 deadline: 1681034204037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,102 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=180 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/eee11ccccf93447988aa9d03baedd504 2023-04-09 09:55:44,114 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=180 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/ddb2726daf21410899618ccb945e8202 2023-04-09 09:55:44,114 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/a4e36e418b3e4ff98855327e7780a1cc as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/a4e36e418b3e4ff98855327e7780a1cc 2023-04-09 09:55:44,117 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/0d414232275a4e2090d78add3f63b5e5 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/0d414232275a4e2090d78add3f63b5e5 2023-04-09 09:55:44,119 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2764d7f8155e1e5b0d641eebe063fa00/A of 2764d7f8155e1e5b0d641eebe063fa00 into a4e36e418b3e4ff98855327e7780a1cc(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:55:44,119 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2764d7f8155e1e5b0d641eebe063fa00: 2023-04-09 09:55:44,119 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00., storeName=2764d7f8155e1e5b0d641eebe063fa00/A, priority=13, startTime=1681034143689; duration=0sec 2023-04-09 09:55:44,119 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:55:44,119 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-09 09:55:44,120 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36148 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-09 09:55:44,120 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1912): 2764d7f8155e1e5b0d641eebe063fa00/C is initiating minor compaction (all files) 2023-04-09 09:55:44,120 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2764d7f8155e1e5b0d641eebe063fa00/C in TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:44,121 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/da297746c817402fbc14e6e570d0e6ca, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/d8db39d6e8e742f4bb39934ff3fd1d87, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/1f67a71e84ee44f9b2c3ef1a6326a073] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp, totalSize=35.3 K 2023-04-09 09:55:44,121 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting da297746c817402fbc14e6e570d0e6ca, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1681034142291 2023-04-09 09:55:44,121 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2764d7f8155e1e5b0d641eebe063fa00/B of 2764d7f8155e1e5b0d641eebe063fa00 into 0d414232275a4e2090d78add3f63b5e5(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:55:44,121 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2764d7f8155e1e5b0d641eebe063fa00: 2023-04-09 09:55:44,121 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00., storeName=2764d7f8155e1e5b0d641eebe063fa00/B, priority=13, startTime=1681034143689; duration=0sec 2023-04-09 09:55:44,121 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:55:44,122 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting d8db39d6e8e742f4bb39934ff3fd1d87, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=132, earliestPutTs=1681034142619 2023-04-09 09:55:44,122 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 1f67a71e84ee44f9b2c3ef1a6326a073, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=145, earliestPutTs=1681034143244 2023-04-09 09:55:44,126 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=180 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/be8fa7f8c284429fad8e59f920b854f5 2023-04-09 09:55:44,127 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2764d7f8155e1e5b0d641eebe063fa00#C#compaction#452 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:55:44,131 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/eee11ccccf93447988aa9d03baedd504 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/eee11ccccf93447988aa9d03baedd504 2023-04-09 09:55:44,134 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/0b65f9bd30b549578d159dbf6a47ca3e as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/0b65f9bd30b549578d159dbf6a47ca3e 2023-04-09 09:55:44,134 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/eee11ccccf93447988aa9d03baedd504, entries=150, sequenceid=180, filesize=11.8 K 2023-04-09 09:55:44,135 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/ddb2726daf21410899618ccb945e8202 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/ddb2726daf21410899618ccb945e8202 2023-04-09 09:55:44,139 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/ddb2726daf21410899618ccb945e8202, entries=150, sequenceid=180, filesize=11.8 K 2023-04-09 09:55:44,140 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2764d7f8155e1e5b0d641eebe063fa00/C of 2764d7f8155e1e5b0d641eebe063fa00 into 0b65f9bd30b549578d159dbf6a47ca3e(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:55:44,140 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2764d7f8155e1e5b0d641eebe063fa00: 2023-04-09 09:55:44,140 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00., storeName=2764d7f8155e1e5b0d641eebe063fa00/C, priority=13, startTime=1681034143689; duration=0sec 2023-04-09 09:55:44,140 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/be8fa7f8c284429fad8e59f920b854f5 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/be8fa7f8c284429fad8e59f920b854f5 2023-04-09 09:55:44,140 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:55:44,143 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/be8fa7f8c284429fad8e59f920b854f5, entries=150, sequenceid=180, filesize=11.8 K 2023-04-09 09:55:44,143 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~370.88 KB/379776, currentSize=53.67 KB/54960 for 2764d7f8155e1e5b0d641eebe063fa00 in 450ms, sequenceid=180, compaction requested=false 2023-04-09 09:55:44,143 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2764d7f8155e1e5b0d641eebe063fa00: 2023-04-09 09:55:44,143 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:44,143 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:44,143 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:44,143 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,144 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:44,144 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:44,145 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,145 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:44,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:44,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:44,146 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:44,146 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:44,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:44,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:44,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:44,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:44,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:44,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@307c5f95[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:44,147 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:44,147 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:44,148 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:44,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:44,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:44,148 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:44,148 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:44,148 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,148 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:44,148 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:44,151 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,151 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:44,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:44,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:44,151 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:44,151 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:44,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:44,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:44,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:44,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:44,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:44,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:44,153 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:44,153 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:44,153 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:44,153 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:44,153 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:44,153 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:44,154 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:44,154 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:44,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:44,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:44,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:44,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:44,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:44,154 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:44,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:44,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:44,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:44,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:44,155 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:44,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:44,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:44,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,156 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:44,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:44,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,158 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:44,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:44,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:44,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:44,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:44,158 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:44,158 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:44,158 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,158 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:44,158 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:44,158 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,158 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:44,159 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:44,159 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:44,158 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:44,158 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:44,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:44,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:44,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 2764d7f8155e1e5b0d641eebe063fa00 2023-04-09 09:55:44,237 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2764d7f8155e1e5b0d641eebe063fa00 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-09 09:55:44,237 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=A 2023-04-09 09:55:44,237 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:44,237 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=B 2023-04-09 09:55:44,237 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:44,237 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=C 2023-04-09 09:55:44,237 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:44,258 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:44,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:44,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:44,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:44,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:44,268 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:44,268 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:44,268 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:44,268 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:44,268 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:44,268 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:44,269 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:44,269 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:44,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:44,270 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,270 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:44,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:44,270 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:44,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:44,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:44,271 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:44,276 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:44,276 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:44,276 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:44,276 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:44,276 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:44,276 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:44,276 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:44,276 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:44,276 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:44,276 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. as already flushing 2023-04-09 09:55:44,276 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:44,277 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:44,277 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:44,277 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,277 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:44,277 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:44,281 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,281 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:44,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:44,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:44,281 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:44,281 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:44,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:44,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:44,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:44,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:44,283 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:44,283 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4981a7d3[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:44,283 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:44,283 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:44,283 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:44,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59302 deadline: 1681034204274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,284 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:44,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:44,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:44,285 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:44,285 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:44,285 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:44,285 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,285 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:44,286 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681034144271, max timestamp=1681034144283Num uniques 859; 2023-04-09 09:55:44,286 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-09 09:55:44,286 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681034144271, max timestamp=1681034144283Num uniques 859; 2023-04-09 09:55:44,286 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-09 09:55:44,286 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681034144271, max timestamp=1681034144283Num uniques 859; 2023-04-09 09:55:44,286 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-09 09:55:44,289 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,289 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:44,289 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:44,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:44,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:44,289 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:44,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:44,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:44,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:44,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:44,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:44,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:44,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:44,292 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:44,292 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:44,292 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:44,292 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:44,292 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:44,294 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:44,294 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:44,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:44,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:44,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:44,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:44,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:44,294 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:44,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:44,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:44,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:44,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:44,295 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:44,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:44,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:44,296 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:44,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:44,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,298 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:44,298 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,298 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:44,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:44,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:44,298 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:44,299 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:44,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:44,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:44,298 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:44,299 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:44,299 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:44,299 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:44,299 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:44,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:44,299 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:44,299 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:44,299 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:44,322 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:44,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59316 deadline: 1681034204316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,328 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:44,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59348 deadline: 1681034204317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,328 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:44,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59344 deadline: 1681034204318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,389 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:44,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59302 deadline: 1681034204385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,399 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:44,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:44,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:44,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:44,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:44,403 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:44,403 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:44,403 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:44,403 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:44,403 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:44,403 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:44,405 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:44,405 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:44,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:44,406 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:44,406 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:44,406 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:44,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:44,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:44,407 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:44,407 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:44,407 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:44,407 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:44,407 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:44,407 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:44,408 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:44,408 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:44,408 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:44,408 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:44,408 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. as already flushing 2023-04-09 09:55:44,408 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:44,408 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:44,408 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:44,408 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,408 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:44,408 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:44,409 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,409 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:44,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:44,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:44,410 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:44,410 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:44,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:44,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:44,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:44,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:44,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:44,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4418eb23[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:44,413 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:44,413 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:44,414 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:44,414 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,414 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:44,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:44,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:44,415 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:44,415 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:44,415 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:44,416 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,416 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:44,416 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:44,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:44,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:44,416 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:44,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:44,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:44,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:44,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:44,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:44,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:44,419 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:44,419 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:44,419 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:44,419 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:44,419 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:44,419 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:44,422 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:44,422 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:44,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:44,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:44,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:44,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:44,422 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:44,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:44,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:44,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:44,423 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:44,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:44,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:44,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:44,424 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:44,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:44,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:44,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,426 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:44,426 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:44,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:44,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:44,427 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,427 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:44,427 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:44,427 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:44,427 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,427 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:44,427 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:44,427 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:44,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:44,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:44,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:44,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:44,427 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:44,427 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:44,428 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:44,429 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:44,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59316 deadline: 1681034204423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,433 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:44,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59348 deadline: 1681034204429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,434 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:44,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59344 deadline: 1681034204429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,436 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:44,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59328 deadline: 1681034204431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,527 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:44,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:44,529 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:44,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:44,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:44,531 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:44,531 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:44,531 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:44,531 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:44,531 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:44,531 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:44,532 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:44,532 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:44,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:44,533 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,533 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:44,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:44,533 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:44,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:44,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:44,534 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:44,534 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:44,534 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:44,534 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:44,534 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:44,534 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:44,536 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:44,536 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:44,536 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:44,536 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:44,536 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. as already flushing 2023-04-09 09:55:44,536 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:44,536 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:44,536 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:44,536 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,536 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:44,536 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:44,538 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:44,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:44,539 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:44,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:44,540 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:44,540 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:44,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:44,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:44,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:44,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:44,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50207b04[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:44,542 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:44,542 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:44,545 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,545 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:44,546 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:44,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:44,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:44,546 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:44,546 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:44,546 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:44,547 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:44,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:44,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:44,548 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:44,548 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:44,548 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:44,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:44,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:44,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:44,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:44,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:44,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:44,555 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:44,555 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:44,555 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:44,556 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:44,556 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:44,557 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:44,557 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:44,557 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:44,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:44,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:44,558 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:44,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:44,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:44,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:44,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:44,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:44,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:44,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:44,559 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:44,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:44,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:44,561 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,561 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:44,561 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:44,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:44,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:44,561 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:44,561 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:44,561 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,561 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:44,561 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:44,561 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:44,562 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:44,562 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:44,561 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:44,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:44,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:44,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:44,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:44,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:44,570 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:44,592 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:44,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59302 deadline: 1681034204590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,639 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:44,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59316 deadline: 1681034204630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,640 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:44,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59344 deadline: 1681034204634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,640 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:44,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59348 deadline: 1681034204635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,662 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:44,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:44,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:44,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:44,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:44,665 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:44,665 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:44,665 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:44,665 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:44,665 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:44,665 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:44,667 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,667 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:44,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:44,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:44,667 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,667 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:44,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:44,667 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:44,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:44,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:44,668 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:44,668 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:44,668 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:44,668 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:44,668 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:44,668 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:44,668 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:44,668 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:44,668 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:44,668 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:44,668 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. as already flushing 2023-04-09 09:55:44,668 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:44,668 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:44,668 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:44,668 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,669 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:44,669 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:44,670 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:44,670 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:44,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:44,670 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:44,670 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:44,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:44,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:44,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:44,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:44,671 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:44,671 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f21662[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:44,671 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:44,671 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:44,673 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:44,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:44,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:44,674 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:44,674 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:44,674 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:44,674 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,674 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:44,675 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:44,675 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,675 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:44,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:44,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:44,675 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:44,675 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=195 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/b42216dee8dd451080de07cef69f2083 2023-04-09 09:55:44,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:44,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:44,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:44,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:44,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:44,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:44,677 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:44,677 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:44,678 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:44,678 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:44,678 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:44,678 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:44,682 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:44,682 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:44,682 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:44,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:44,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:44,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:44,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:44,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:44,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:44,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:44,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:44,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:44,683 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:44,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:44,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:44,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,685 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:44,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:44,685 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=195 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/5f4ec3e0cdb24f5f897a8e578811d26b 2023-04-09 09:55:44,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,688 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,689 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:44,689 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:44,689 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:44,689 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:44,689 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:44,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:44,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:44,689 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:44,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:44,689 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:44,689 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:44,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:44,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:44,689 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:44,689 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:44,689 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:44,789 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:44,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:44,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:44,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:44,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:44,792 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:44,792 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:44,792 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:44,792 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:44,793 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:44,793 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:44,794 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,794 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:44,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:44,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:44,794 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,794 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:44,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:44,794 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:44,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:44,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:44,795 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:44,795 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:44,795 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:44,795 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:44,795 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:44,795 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:44,796 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:44,796 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:44,796 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:44,796 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:44,796 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. as already flushing 2023-04-09 09:55:44,796 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:44,796 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:44,796 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:44,796 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,796 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:44,796 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:44,799 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:44,799 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:44,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:44,800 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:44,800 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:44,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:44,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:44,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:44,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:44,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:44,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@460986d0[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:44,801 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:44,801 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:44,802 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:44,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:44,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:44,802 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:44,802 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:44,802 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,802 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:44,802 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:44,803 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:44,803 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:44,803 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,803 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:44,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:44,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:44,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:44,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:44,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:44,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:44,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:44,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:44,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:44,805 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:44,806 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:44,806 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:44,806 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:44,806 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:44,807 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:44,807 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:44,807 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:44,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:44,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:44,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:44,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:44,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:44,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:44,808 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:44,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:44,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:44,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:44,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:44,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:44,809 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:44,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:44,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,811 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,811 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:44,812 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:44,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:44,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:44,811 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:44,812 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:44,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:44,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:44,812 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:44,812 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:44,812 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:44,812 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,812 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:44,812 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:44,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:44,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:44,812 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:44,812 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:44,898 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:44,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59302 deadline: 1681034204895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,912 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:44,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:44,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:44,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:44,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:44,915 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:44,915 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:44,915 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:44,915 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:44,916 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:44,916 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:44,917 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,917 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:44,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:44,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:44,917 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,917 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:44,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:44,918 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:44,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:44,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:44,918 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:44,918 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:44,918 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:44,919 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:44,919 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:44,919 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:44,919 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:44,919 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:44,919 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:44,919 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:44,919 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. as already flushing 2023-04-09 09:55:44,919 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:44,919 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:44,919 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:44,919 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,919 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:44,919 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:44,921 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,921 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:44,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:44,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:44,921 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:44,921 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:44,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:44,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:44,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:44,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:44,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:44,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45cb2cc1[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:44,922 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:44,922 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:44,923 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:44,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:44,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:44,924 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:44,924 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,924 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:44,924 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:44,924 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:44,926 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:44,926 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:44,926 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,926 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:44,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:44,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:44,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:44,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:44,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:44,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:44,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:44,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:44,929 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:44,929 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:44,929 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:44,929 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:44,929 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:44,929 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:44,930 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:44,930 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:44,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:44,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:44,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:44,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:44,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:44,930 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:44,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:44,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:44,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:44,931 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:44,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:44,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:44,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:44,932 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:44,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:44,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,934 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,934 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:44,934 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:44,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:44,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:44,934 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:44,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:44,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:44,934 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:44,934 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:44,934 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,934 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:44,934 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:44,934 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:44,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:44,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:44,934 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:44,934 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:44,935 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:44,946 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:44,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59316 deadline: 1681034204940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,946 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:44,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59344 deadline: 1681034204941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:44,947 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:44,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59348 deadline: 1681034204942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:45,035 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:45,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:45,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:45,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:45,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:45,039 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:45,039 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:45,039 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:45,039 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:45,039 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:45,039 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:45,040 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:45,040 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:45,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:45,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:45,041 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:45,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:45,041 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:45,041 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:45,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:45,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:45,041 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:45,041 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:45,041 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:45,041 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:45,042 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:45,042 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:45,042 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:45,042 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:45,042 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:45,042 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:45,042 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. as already flushing 2023-04-09 09:55:45,042 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:45,042 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:45,042 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:45,042 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:45,042 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:45,042 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:45,045 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:45,046 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:45,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:45,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:45,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:45,046 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:45,046 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:45,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:45,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:45,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:45,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:45,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:45,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:45,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16e08727[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:45,047 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:45,047 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:45,048 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:45,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:45,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:45,048 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:45,048 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:45,048 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:45,048 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:45,048 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:45,050 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:45,050 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:45,050 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:45,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:45,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:45,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:45,050 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:45,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:45,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:45,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:45,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:45,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:45,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:45,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:45,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:45,052 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:45,052 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:45,052 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:45,052 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:45,052 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:45,052 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:45,054 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:45,054 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:45,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:45,054 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:45,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:45,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:45,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:45,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:45,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:45,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:45,054 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:45,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:45,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:45,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:45,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:45,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:45,055 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:45,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:45,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:45,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:45,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:45,060 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:45,060 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:45,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:45,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:45,060 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:45,060 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:45,060 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:45,060 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:45,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:45,065 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:45,060 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:45,065 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:45,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:45,065 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:45,065 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:45,065 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:45,065 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:45,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:45,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:45,094 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=195 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/d07a76ec0b72488bb7aa3d89eeba0ab8 2023-04-09 09:55:45,097 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/b42216dee8dd451080de07cef69f2083 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/b42216dee8dd451080de07cef69f2083 2023-04-09 09:55:45,100 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/b42216dee8dd451080de07cef69f2083, entries=150, sequenceid=195, filesize=11.8 K 2023-04-09 09:55:45,100 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/5f4ec3e0cdb24f5f897a8e578811d26b as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/5f4ec3e0cdb24f5f897a8e578811d26b 2023-04-09 09:55:45,103 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/5f4ec3e0cdb24f5f897a8e578811d26b, entries=150, sequenceid=195, filesize=11.8 K 2023-04-09 09:55:45,104 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/d07a76ec0b72488bb7aa3d89eeba0ab8 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/d07a76ec0b72488bb7aa3d89eeba0ab8 2023-04-09 09:55:45,107 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/d07a76ec0b72488bb7aa3d89eeba0ab8, entries=150, sequenceid=195, filesize=11.8 K 2023-04-09 09:55:45,107 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.76 KB/123660 for 2764d7f8155e1e5b0d641eebe063fa00 in 870ms, sequenceid=195, compaction requested=true 2023-04-09 09:55:45,108 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2764d7f8155e1e5b0d641eebe063fa00: 2023-04-09 09:55:45,108 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:55:45,108 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:55:45,108 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-09 09:55:45,108 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-09 09:55:45,108 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:55:45,109 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36450 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-09 09:55:45,109 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1912): 2764d7f8155e1e5b0d641eebe063fa00/A is initiating minor compaction (all files) 2023-04-09 09:55:45,109 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36450 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-09 09:55:45,109 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2764d7f8155e1e5b0d641eebe063fa00/A in TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:45,109 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): 2764d7f8155e1e5b0d641eebe063fa00/B is initiating minor compaction (all files) 2023-04-09 09:55:45,109 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/a4e36e418b3e4ff98855327e7780a1cc, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/eee11ccccf93447988aa9d03baedd504, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/b42216dee8dd451080de07cef69f2083] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp, totalSize=35.6 K 2023-04-09 09:55:45,109 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2764d7f8155e1e5b0d641eebe063fa00/B in TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:45,109 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/0d414232275a4e2090d78add3f63b5e5, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/ddb2726daf21410899618ccb945e8202, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/5f4ec3e0cdb24f5f897a8e578811d26b] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp, totalSize=35.6 K 2023-04-09 09:55:45,109 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting a4e36e418b3e4ff98855327e7780a1cc, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=145, earliestPutTs=1681034143244 2023-04-09 09:55:45,110 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 0d414232275a4e2090d78add3f63b5e5, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=145, earliestPutTs=1681034143244 2023-04-09 09:55:45,110 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting eee11ccccf93447988aa9d03baedd504, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=180, earliestPutTs=1681034143292 2023-04-09 09:55:45,110 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting ddb2726daf21410899618ccb945e8202, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=180, earliestPutTs=1681034143292 2023-04-09 09:55:45,110 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting b42216dee8dd451080de07cef69f2083, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1681034143916 2023-04-09 09:55:45,110 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 5f4ec3e0cdb24f5f897a8e578811d26b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1681034143916 2023-04-09 09:55:45,115 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2764d7f8155e1e5b0d641eebe063fa00#A#compaction#456 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:55:45,116 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): 2764d7f8155e1e5b0d641eebe063fa00#B#compaction#457 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:55:45,165 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:45,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:45,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:45,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:45,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:45,168 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:45,169 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:45,169 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:45,169 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:45,169 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:45,169 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:45,170 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:45,170 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:45,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:45,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:45,171 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:45,171 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:45,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:45,171 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:45,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:45,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:45,172 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:45,172 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:45,172 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:45,172 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:45,172 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:45,172 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:45,173 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:45,173 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:45,173 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:45,174 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:45,174 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2764d7f8155e1e5b0d641eebe063fa00 3/3 column families, dataSize=120.76 KB heapSize=340.45 KB 2023-04-09 09:55:45,174 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=A 2023-04-09 09:55:45,174 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:55:45,174 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=B 2023-04-09 09:55:45,174 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:55:45,174 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=C 2023-04-09 09:55:45,174 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:55:45,185 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=229 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/66d87d295c4544f68f2138fbae5281ee 2023-04-09 09:55:45,195 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=229 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/8e0c0b9dc5214f308c23441c5fc925ee 2023-04-09 09:55:45,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 2764d7f8155e1e5b0d641eebe063fa00 2023-04-09 09:55:45,406 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. as already flushing 2023-04-09 09:55:45,455 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:45,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59328 deadline: 1681034205446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:45,457 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:45,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59344 deadline: 1681034205447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:45,457 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:45,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59302 deadline: 1681034205447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:45,457 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:45,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59348 deadline: 1681034205448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:45,457 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:45,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59316 deadline: 1681034205450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:45,524 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/fab256d1416f402f855921854ae78b47 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/fab256d1416f402f855921854ae78b47 2023-04-09 09:55:45,526 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/0f037ba4104c4f12a69a41ac4f3f34d5 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/0f037ba4104c4f12a69a41ac4f3f34d5 2023-04-09 09:55:45,529 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2764d7f8155e1e5b0d641eebe063fa00/B of 2764d7f8155e1e5b0d641eebe063fa00 into fab256d1416f402f855921854ae78b47(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:55:45,529 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2764d7f8155e1e5b0d641eebe063fa00: 2023-04-09 09:55:45,529 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00., storeName=2764d7f8155e1e5b0d641eebe063fa00/B, priority=13, startTime=1681034145108; duration=0sec 2023-04-09 09:55:45,529 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:55:45,529 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-09 09:55:45,530 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2764d7f8155e1e5b0d641eebe063fa00/A of 2764d7f8155e1e5b0d641eebe063fa00 into 0f037ba4104c4f12a69a41ac4f3f34d5(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:55:45,530 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36450 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-09 09:55:45,530 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2764d7f8155e1e5b0d641eebe063fa00: 2023-04-09 09:55:45,530 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00., storeName=2764d7f8155e1e5b0d641eebe063fa00/A, priority=13, startTime=1681034145108; duration=0sec 2023-04-09 09:55:45,530 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): 2764d7f8155e1e5b0d641eebe063fa00/C is initiating minor compaction (all files) 2023-04-09 09:55:45,530 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:55:45,530 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2764d7f8155e1e5b0d641eebe063fa00/C in TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:45,530 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/0b65f9bd30b549578d159dbf6a47ca3e, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/be8fa7f8c284429fad8e59f920b854f5, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/d07a76ec0b72488bb7aa3d89eeba0ab8] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp, totalSize=35.6 K 2023-04-09 09:55:45,531 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 0b65f9bd30b549578d159dbf6a47ca3e, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=145, earliestPutTs=1681034143244 2023-04-09 09:55:45,531 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting be8fa7f8c284429fad8e59f920b854f5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=180, earliestPutTs=1681034143292 2023-04-09 09:55:45,531 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting d07a76ec0b72488bb7aa3d89eeba0ab8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1681034143916 2023-04-09 09:55:45,536 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): 2764d7f8155e1e5b0d641eebe063fa00#C#compaction#461 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:55:45,542 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/9c35607a98664a6bb5ad65b69d70ea8f as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/9c35607a98664a6bb5ad65b69d70ea8f 2023-04-09 09:55:45,545 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2764d7f8155e1e5b0d641eebe063fa00/C of 2764d7f8155e1e5b0d641eebe063fa00 into 9c35607a98664a6bb5ad65b69d70ea8f(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:55:45,545 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2764d7f8155e1e5b0d641eebe063fa00: 2023-04-09 09:55:45,545 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00., storeName=2764d7f8155e1e5b0d641eebe063fa00/C, priority=13, startTime=1681034145108; duration=0sec 2023-04-09 09:55:45,545 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:55:45,562 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:45,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59328 deadline: 1681034205557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:45,563 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:45,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59302 deadline: 1681034205558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:45,604 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=229 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/e8af8ded416944b0852dd00cdc61689b 2023-04-09 09:55:45,608 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/66d87d295c4544f68f2138fbae5281ee as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/66d87d295c4544f68f2138fbae5281ee 2023-04-09 09:55:45,611 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/66d87d295c4544f68f2138fbae5281ee, entries=150, sequenceid=229, filesize=11.8 K 2023-04-09 09:55:45,611 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/8e0c0b9dc5214f308c23441c5fc925ee as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/8e0c0b9dc5214f308c23441c5fc925ee 2023-04-09 09:55:45,615 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/8e0c0b9dc5214f308c23441c5fc925ee, entries=150, sequenceid=229, filesize=11.8 K 2023-04-09 09:55:45,615 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/e8af8ded416944b0852dd00cdc61689b as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/e8af8ded416944b0852dd00cdc61689b 2023-04-09 09:55:45,618 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/e8af8ded416944b0852dd00cdc61689b, entries=150, sequenceid=229, filesize=11.8 K 2023-04-09 09:55:45,619 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~340.41 KB/348576, currentSize=60.38 KB/61830 for 2764d7f8155e1e5b0d641eebe063fa00 in 445ms, sequenceid=229, compaction requested=false 2023-04-09 09:55:45,619 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2764d7f8155e1e5b0d641eebe063fa00: 2023-04-09 09:55:45,619 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:45,619 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:45,619 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:45,619 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:45,619 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:45,619 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:45,621 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:45,621 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:45,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:45,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:45,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:45,621 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:45,621 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:45,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:45,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:45,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:45,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:45,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:45,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:45,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b3d29f[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:45,623 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:45,623 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:45,625 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:45,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:45,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:45,625 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:45,625 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:45,625 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:45,625 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:45,626 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:45,626 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:45,626 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:45,627 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:45,627 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:45,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:45,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:45,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:45,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:45,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:45,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:45,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:45,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:45,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:45,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:45,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:45,629 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:45,629 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:45,629 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:45,629 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:45,629 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:45,629 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:45,630 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:45,630 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:45,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:45,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:45,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:45,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:45,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:45,630 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:45,631 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:45,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:45,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:45,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:45,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:45,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:45,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:45,632 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:45,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:45,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:45,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:45,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:45,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:45,634 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:45,634 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:45,634 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:45,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:45,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:45,634 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:45,634 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:45,634 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:45,634 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:45,634 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:45,634 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:45,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:45,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:45,634 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:45,634 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:45,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:45,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:45,635 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:45,635 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:45,735 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:45,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:45,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:45,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:45,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:45,744 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:45,744 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:45,744 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:45,744 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:45,745 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:45,745 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:45,747 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:45,747 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:45,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:45,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:45,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:45,747 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:45,747 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:45,747 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:45,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:45,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:45,748 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:45,751 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:45,751 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:45,751 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:45,751 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:45,751 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:45,759 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:45,759 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:45,759 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:45,759 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:45,759 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2764d7f8155e1e5b0d641eebe063fa00 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-09 09:55:45,760 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=A 2023-04-09 09:55:45,760 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:45,760 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=B 2023-04-09 09:55:45,760 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:45,760 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=C 2023-04-09 09:55:45,760 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:45,766 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. as already flushing 2023-04-09 09:55:45,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 2764d7f8155e1e5b0d641eebe063fa00 2023-04-09 09:55:45,816 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:45,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59328 deadline: 1681034205807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:45,816 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:45,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59302 deadline: 1681034205810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:45,922 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:45,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59328 deadline: 1681034205917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:45,922 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:45,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59302 deadline: 1681034205917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:46,124 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:46,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59328 deadline: 1681034206123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:46,127 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:46,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59302 deadline: 1681034206124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:46,177 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=244 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/76aa650003c247beb83aea42f5f03e61 2023-04-09 09:55:46,187 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=244 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/f2aed27732d4414abfb26cbe4c395202 2023-04-09 09:55:46,429 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:46,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59328 deadline: 1681034206427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:46,433 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:46,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59302 deadline: 1681034206430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:46,460 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:46,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59316 deadline: 1681034206458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:46,463 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:46,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59344 deadline: 1681034206459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:46,466 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:46,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59348 deadline: 1681034206463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:46,596 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=244 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/647c66655efd4bd9a7e8d1be179d7293 2023-04-09 09:55:46,600 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/76aa650003c247beb83aea42f5f03e61 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/76aa650003c247beb83aea42f5f03e61 2023-04-09 09:55:46,603 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/76aa650003c247beb83aea42f5f03e61, entries=150, sequenceid=244, filesize=11.8 K 2023-04-09 09:55:46,604 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/f2aed27732d4414abfb26cbe4c395202 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/f2aed27732d4414abfb26cbe4c395202 2023-04-09 09:55:46,607 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/f2aed27732d4414abfb26cbe4c395202, entries=150, sequenceid=244, filesize=11.8 K 2023-04-09 09:55:46,607 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/647c66655efd4bd9a7e8d1be179d7293 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/647c66655efd4bd9a7e8d1be179d7293 2023-04-09 09:55:46,611 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/647c66655efd4bd9a7e8d1be179d7293, entries=150, sequenceid=244, filesize=11.8 K 2023-04-09 09:55:46,611 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 2764d7f8155e1e5b0d641eebe063fa00 in 852ms, sequenceid=244, compaction requested=true 2023-04-09 09:55:46,611 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2764d7f8155e1e5b0d641eebe063fa00: 2023-04-09 09:55:46,611 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:46,611 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:46,612 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:46,612 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:46,612 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:46,612 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:46,613 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:46,613 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:46,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:46,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:46,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:46,613 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:46,613 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:46,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:46,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:46,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:46,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:46,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:46,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:46,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e06fbdf[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:46,615 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:46,615 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:46,616 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:46,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:46,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:46,616 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:46,616 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:46,616 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:46,616 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:46,616 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:46,617 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:46,617 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:46,617 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:46,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:46,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:46,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:46,617 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:46,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:46,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:46,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:46,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:46,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:46,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:46,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:46,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:46,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:46,619 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:46,620 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:46,620 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:46,620 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:46,620 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:46,622 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:46,622 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:46,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:46,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:46,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:46,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:46,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:46,622 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:46,622 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:46,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:46,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:46,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:46,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:46,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:46,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:46,623 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:46,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:46,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:46,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:46,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:46,625 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:46,625 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:46,625 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:46,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:46,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:46,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:46,625 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:46,626 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:46,626 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:46,626 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:46,626 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:46,625 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:46,626 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:46,626 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:46,626 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:46,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:46,626 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:46,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:46,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:46,726 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:46,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:46,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:46,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:46,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:46,729 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:46,729 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:46,729 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:46,729 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:46,729 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:46,729 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:46,730 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:46,730 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:46,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:46,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:46,731 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:46,731 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:46,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:46,731 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:46,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:46,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:46,732 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:46,732 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:46,732 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:46,732 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:46,732 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:46,732 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:46,732 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:46,732 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:46,732 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:46,732 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:46,732 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2764d7f8155e1e5b0d641eebe063fa00 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-09 09:55:46,733 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=A 2023-04-09 09:55:46,733 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:46,733 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=B 2023-04-09 09:55:46,733 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:46,733 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=C 2023-04-09 09:55:46,733 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:46,740 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=264 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/b09bec28124a47038a426d9d757d1e2b 2023-04-09 09:55:46,762 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=264 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/65ce40a0eab549dc8d5d1a447ae7ab8c 2023-04-09 09:55:46,782 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=264 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/b0ee312e0dad46909d8690187b0246b4 2023-04-09 09:55:46,786 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/b09bec28124a47038a426d9d757d1e2b as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/b09bec28124a47038a426d9d757d1e2b 2023-04-09 09:55:46,789 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/b09bec28124a47038a426d9d757d1e2b, entries=150, sequenceid=264, filesize=11.9 K 2023-04-09 09:55:46,789 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/65ce40a0eab549dc8d5d1a447ae7ab8c as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/65ce40a0eab549dc8d5d1a447ae7ab8c 2023-04-09 09:55:46,793 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/65ce40a0eab549dc8d5d1a447ae7ab8c, entries=150, sequenceid=264, filesize=11.9 K 2023-04-09 09:55:46,793 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/b0ee312e0dad46909d8690187b0246b4 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/b0ee312e0dad46909d8690187b0246b4 2023-04-09 09:55:46,797 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/b0ee312e0dad46909d8690187b0246b4, entries=150, sequenceid=264, filesize=11.9 K 2023-04-09 09:55:46,798 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for 2764d7f8155e1e5b0d641eebe063fa00 in 66ms, sequenceid=264, compaction requested=true 2023-04-09 09:55:46,798 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2764d7f8155e1e5b0d641eebe063fa00: 2023-04-09 09:55:46,798 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:46,798 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:46,798 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:46,798 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:46,798 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:46,798 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:46,800 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:46,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:46,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:46,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:46,800 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:46,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:46,800 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:46,800 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:46,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:46,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:46,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:46,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:46,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:46,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12b773fa[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:46,801 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:46,801 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:46,804 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:46,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:46,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:46,804 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:46,804 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:46,804 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:46,804 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:46,804 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:46,805 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:46,805 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:46,805 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:46,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:46,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:46,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:46,805 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:46,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:46,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:46,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:46,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:46,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:46,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:46,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:46,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:46,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:46,807 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:46,807 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:46,807 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:46,808 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:46,808 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:46,809 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:46,809 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:46,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:46,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:46,809 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:46,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:46,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:46,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:46,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:46,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:46,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:46,810 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:46,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:46,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:46,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:46,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:46,812 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:46,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:46,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:46,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:46,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:46,813 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:46,813 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:46,813 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:46,813 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:46,813 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:46,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:46,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:46,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:46,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:46,813 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:46,813 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:46,814 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:46,814 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:46,813 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:46,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:46,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:46,813 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:46,814 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:46,814 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:46,914 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:46,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:46,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:46,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:46,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:46,917 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:46,917 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:46,917 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:46,917 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:46,917 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:46,917 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:46,922 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:46,922 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:46,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:46,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:46,923 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:46,923 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:46,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:46,923 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:46,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:46,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:46,923 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:46,924 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:46,924 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:46,924 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:46,924 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:46,924 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:46,924 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:46,924 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:46,924 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:46,924 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:46,924 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2764d7f8155e1e5b0d641eebe063fa00: 2023-04-09 09:55:46,924 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:46,924 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:46,924 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:46,924 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:46,924 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:46,924 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:46,926 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:46,926 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:46,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:46,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:46,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:46,926 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:46,926 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:46,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:46,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:46,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:46,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:46,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:46,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:46,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@fc1734[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:46,927 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:46,927 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:46,928 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:46,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:46,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:46,928 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:46,928 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:46,928 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:46,928 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:46,928 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:46,930 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:46,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:46,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:46,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:46,930 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:46,931 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:46,931 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:46,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:46,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:46,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:46,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:46,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:46,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:46,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:46,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:46,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:46,934 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:46,934 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:46,934 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:46,934 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:46,934 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:46,935 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:46,935 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:46,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:46,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:46,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:46,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:46,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:46,935 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:46,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:46,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:46,936 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:46,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:46,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:46,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:46,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:46,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:46,938 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:46,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:46,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:46,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:46,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:46,940 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:46,940 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:46,940 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:46,940 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:46,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:46,940 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:46,940 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:46,940 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:46,940 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:46,940 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:46,940 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:46,940 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:46,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:46,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:46,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:46,941 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:46,941 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:46,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:46,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:46,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 2764d7f8155e1e5b0d641eebe063fa00 2023-04-09 09:55:46,947 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2764d7f8155e1e5b0d641eebe063fa00 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-09 09:55:46,948 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=A 2023-04-09 09:55:46,948 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:46,948 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=B 2023-04-09 09:55:46,948 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:46,948 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=C 2023-04-09 09:55:46,948 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:47,028 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681034147008, max timestamp=1681034147027Num uniques 859; 2023-04-09 09:55:47,029 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-09 09:55:47,029 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681034147008, max timestamp=1681034147027Num uniques 859; 2023-04-09 09:55:47,029 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681034147008, max timestamp=1681034147027Num uniques 859; 2023-04-09 09:55:47,029 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-09 09:55:47,029 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-09 09:55:47,041 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:47,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:47,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:47,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:47,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:47,048 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:47,048 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:47,048 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:47,048 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:47,048 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:47,049 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:47,052 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,052 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:47,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:47,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:47,052 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,052 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:47,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:47,052 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:47,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:47,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:47,053 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:47,053 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:47,053 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:47,053 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:47,053 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:47,053 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:47,053 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:47,053 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:47,053 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:47,053 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:47,053 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. as already flushing 2023-04-09 09:55:47,054 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:47,054 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:47,054 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:47,054 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,054 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:47,054 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:47,056 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,056 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:47,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:47,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:47,056 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:47,056 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:47,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:47,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:47,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:47,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:47,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:47,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ac10a9e[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:47,057 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:47,057 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:47,060 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:47,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:47,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:47,060 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:47,060 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,060 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:47,060 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:47,060 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:47,062 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,062 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:47,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:47,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:47,062 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:47,062 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:47,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:47,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:47,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:47,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:47,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:47,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:47,064 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:47,064 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:47,064 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:47,064 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:47,064 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:47,064 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:47,066 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:47,066 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:47,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:47,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:47,066 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:47,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:47,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:47,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:47,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:47,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:47,067 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:47,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:47,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:47,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:47,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:47,068 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:47,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:47,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,070 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:47,070 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:47,070 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:47,070 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,070 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:47,070 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:47,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:47,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:47,070 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:47,070 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:47,070 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:47,070 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:47,070 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:47,070 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,070 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:47,070 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:47,070 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:47,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:47,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:47,114 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:47,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59328 deadline: 1681034207106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,114 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:47,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59302 deadline: 1681034207106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,170 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:47,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:47,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:47,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:47,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:47,174 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:47,174 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:47,174 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:47,174 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:47,174 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:47,174 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:47,176 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,176 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:47,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:47,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:47,176 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,176 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:47,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:47,176 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:47,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:47,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:47,177 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:47,177 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:47,177 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:47,177 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:47,177 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:47,177 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:47,177 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:47,177 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:47,178 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:47,178 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:47,178 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. as already flushing 2023-04-09 09:55:47,178 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:47,178 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:47,178 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:47,178 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,178 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:47,178 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:47,180 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,180 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:47,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:47,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:47,181 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:47,181 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:47,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:47,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:47,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:47,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:47,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:47,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2320c21a[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:47,182 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:47,182 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:47,183 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:47,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:47,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:47,183 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:47,183 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:47,183 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,183 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:47,183 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:47,185 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,185 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:47,185 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:47,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:47,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:47,185 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:47,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:47,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:47,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:47,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:47,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:47,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:47,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:47,187 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:47,188 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:47,188 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:47,188 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:47,188 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:47,189 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:47,189 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:47,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:47,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:47,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:47,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:47,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:47,189 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:47,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:47,189 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:47,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:47,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:47,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:47,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:47,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:47,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,191 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:47,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:47,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,194 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:47,194 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:47,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:47,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:47,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:47,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:47,194 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,194 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:47,194 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:47,194 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:47,194 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:47,194 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:47,194 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:47,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:47,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:47,194 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,195 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:47,195 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:47,195 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:47,219 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:47,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59328 deadline: 1681034207215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,220 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:47,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59302 deadline: 1681034207215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,295 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:47,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:47,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:47,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:47,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:47,299 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:47,299 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:47,299 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:47,299 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:47,299 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:47,299 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:47,300 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,301 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:47,301 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,301 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:47,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:47,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:47,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:47,303 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:47,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:47,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:47,303 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:47,304 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:47,304 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:47,304 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:47,304 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:47,304 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:47,304 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:47,304 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:47,304 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:47,304 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:47,304 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. as already flushing 2023-04-09 09:55:47,304 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:47,304 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:47,304 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:47,304 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,304 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:47,304 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:47,308 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:47,309 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:47,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:47,309 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:47,309 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:47,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:47,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:47,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:47,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:47,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:47,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@aa15d27[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:47,312 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:47,312 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:47,313 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:47,314 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,314 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:47,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:47,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:47,314 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:47,314 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:47,314 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:47,315 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:47,315 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:47,315 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:47,315 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:47,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:47,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:47,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:47,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:47,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:47,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:47,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:47,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:47,320 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:47,320 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:47,320 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:47,320 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:47,320 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:47,323 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:47,323 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:47,323 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:47,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:47,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:47,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:47,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:47,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:47,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:47,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:47,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:47,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:47,324 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:47,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:47,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:47,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,326 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:47,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:47,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,332 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,332 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:47,332 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:47,332 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,332 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:47,332 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:47,332 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:47,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:47,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:47,332 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:47,332 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:47,332 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:47,332 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:47,332 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:47,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:47,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:47,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:47,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:47,354 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=276 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/cddbefe4674e46ef8c9a2cfcb6c2a74e 2023-04-09 09:55:47,365 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=276 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/fea257912ef94f5382a82508be5892c7 2023-04-09 09:55:47,374 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=276 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/ddac8b65087f438c87b0ac46bf99c85c 2023-04-09 09:55:47,379 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/cddbefe4674e46ef8c9a2cfcb6c2a74e as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/cddbefe4674e46ef8c9a2cfcb6c2a74e 2023-04-09 09:55:47,383 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/cddbefe4674e46ef8c9a2cfcb6c2a74e, entries=250, sequenceid=276, filesize=16.7 K 2023-04-09 09:55:47,384 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/fea257912ef94f5382a82508be5892c7 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/fea257912ef94f5382a82508be5892c7 2023-04-09 09:55:47,387 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/fea257912ef94f5382a82508be5892c7, entries=150, sequenceid=276, filesize=11.9 K 2023-04-09 09:55:47,387 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/ddac8b65087f438c87b0ac46bf99c85c as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/ddac8b65087f438c87b0ac46bf99c85c 2023-04-09 09:55:47,390 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/ddac8b65087f438c87b0ac46bf99c85c, entries=150, sequenceid=276, filesize=11.9 K 2023-04-09 09:55:47,391 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 2764d7f8155e1e5b0d641eebe063fa00 in 444ms, sequenceid=276, compaction requested=true 2023-04-09 09:55:47,391 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2764d7f8155e1e5b0d641eebe063fa00: 2023-04-09 09:55:47,391 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:55:47,391 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-09 09:55:47,391 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-09 09:55:47,391 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:55:47,391 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:55:47,392 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 65804 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-09 09:55:47,393 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60924 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-09 09:55:47,393 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1912): 2764d7f8155e1e5b0d641eebe063fa00/A is initiating minor compaction (all files) 2023-04-09 09:55:47,393 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): 2764d7f8155e1e5b0d641eebe063fa00/B is initiating minor compaction (all files) 2023-04-09 09:55:47,393 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2764d7f8155e1e5b0d641eebe063fa00/A in TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:47,393 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2764d7f8155e1e5b0d641eebe063fa00/B in TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:47,393 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/0f037ba4104c4f12a69a41ac4f3f34d5, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/66d87d295c4544f68f2138fbae5281ee, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/76aa650003c247beb83aea42f5f03e61, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/b09bec28124a47038a426d9d757d1e2b, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/cddbefe4674e46ef8c9a2cfcb6c2a74e] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp, totalSize=64.3 K 2023-04-09 09:55:47,393 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/fab256d1416f402f855921854ae78b47, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/8e0c0b9dc5214f308c23441c5fc925ee, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/f2aed27732d4414abfb26cbe4c395202, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/65ce40a0eab549dc8d5d1a447ae7ab8c, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/fea257912ef94f5382a82508be5892c7] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp, totalSize=59.5 K 2023-04-09 09:55:47,393 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 0f037ba4104c4f12a69a41ac4f3f34d5, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1681034143916 2023-04-09 09:55:47,393 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting fab256d1416f402f855921854ae78b47, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1681034143916 2023-04-09 09:55:47,393 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 66d87d295c4544f68f2138fbae5281ee, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=229, earliestPutTs=1681034144295 2023-04-09 09:55:47,394 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 8e0c0b9dc5214f308c23441c5fc925ee, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=229, earliestPutTs=1681034144295 2023-04-09 09:55:47,394 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 76aa650003c247beb83aea42f5f03e61, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=244, earliestPutTs=1681034145432 2023-04-09 09:55:47,394 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting f2aed27732d4414abfb26cbe4c395202, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=244, earliestPutTs=1681034145432 2023-04-09 09:55:47,394 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting b09bec28124a47038a426d9d757d1e2b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=264, earliestPutTs=1681034145803 2023-04-09 09:55:47,394 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 65ce40a0eab549dc8d5d1a447ae7ab8c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=264, earliestPutTs=1681034145803 2023-04-09 09:55:47,395 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting cddbefe4674e46ef8c9a2cfcb6c2a74e, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=276, earliestPutTs=1681034146940 2023-04-09 09:55:47,395 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting fea257912ef94f5382a82508be5892c7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=276, earliestPutTs=1681034146940 2023-04-09 09:55:47,426 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): 2764d7f8155e1e5b0d641eebe063fa00#B#compaction#471 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:55:47,426 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2764d7f8155e1e5b0d641eebe063fa00#A#compaction#472 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:55:47,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 2764d7f8155e1e5b0d641eebe063fa00 2023-04-09 09:55:47,429 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2764d7f8155e1e5b0d641eebe063fa00 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-09 09:55:47,429 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=A 2023-04-09 09:55:47,429 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:55:47,429 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=B 2023-04-09 09:55:47,429 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:55:47,429 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=C 2023-04-09 09:55:47,429 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:55:47,433 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:47,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:47,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:47,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:47,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:47,448 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:47,448 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:47,448 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:47,448 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:47,449 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:47,449 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:47,450 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,450 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:47,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:47,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:47,450 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,450 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:47,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:47,450 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:47,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:47,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:47,451 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:47,472 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:47,472 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:47,472 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:47,473 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:47,473 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:47,473 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:47,473 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:47,474 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:47,474 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:47,474 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. as already flushing 2023-04-09 09:55:47,474 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:47,474 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:47,474 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:47,474 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,475 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:47,476 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:47,476 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:47,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59302 deadline: 1681034207465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,477 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:47,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59328 deadline: 1681034207465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,478 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,478 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:47,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:47,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:47,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:47,478 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:47,478 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:47,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:47,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:47,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:47,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:47,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a84aef0[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:47,480 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:47,480 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:47,481 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:47,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:47,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:47,481 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:47,481 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:47,481 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:47,481 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,481 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:47,483 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:47,483 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,483 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:47,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:47,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:47,483 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:47,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:47,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:47,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:47,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:47,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:47,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:47,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:47,485 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:47,487 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:47,487 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:47,487 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:47,487 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:47,489 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:47,489 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:47,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:47,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:47,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:47,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:47,489 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:47,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:47,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:47,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:47,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:47,490 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:47,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:47,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:47,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:47,491 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:47,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:47,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,493 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:47,493 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:47,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:47,493 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,494 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:47,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:47,494 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:47,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:47,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:47,494 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:47,494 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,494 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:47,494 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:47,494 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:47,494 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:47,494 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:47,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:47,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:47,494 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:47,585 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:47,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59302 deadline: 1681034207577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,585 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:47,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59328 deadline: 1681034207578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,594 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:47,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:47,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:47,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:47,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:47,597 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:47,597 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:47,598 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:47,598 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:47,598 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:47,598 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:47,599 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,599 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:47,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:47,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:47,599 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,599 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:47,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:47,600 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:47,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:47,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:47,600 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:47,600 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:47,600 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:47,601 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:47,601 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:47,601 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:47,601 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:47,601 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:47,601 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:47,601 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:47,601 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. as already flushing 2023-04-09 09:55:47,601 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:47,601 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:47,601 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:47,601 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,601 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:47,601 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:47,604 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,604 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:47,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:47,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:47,604 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:47,604 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:47,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:47,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:47,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:47,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:47,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:47,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7883ccc4[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:47,606 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:47,606 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:47,607 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:47,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:47,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:47,607 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:47,607 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:47,607 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:47,607 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,607 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:47,608 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:47,608 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:47,608 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,608 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:47,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:47,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:47,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:47,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:47,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:47,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:47,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:47,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:47,611 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:47,611 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:47,611 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:47,611 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:47,611 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:47,611 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:47,612 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:47,612 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:47,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:47,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:47,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:47,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:47,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:47,612 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:47,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:47,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:47,613 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:47,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:47,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:47,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:47,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:47,614 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:47,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:47,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,617 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,618 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:47,618 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:47,618 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:47,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:47,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:47,618 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:47,618 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:47,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:47,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:47,618 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:47,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:47,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:47,618 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,618 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:47,618 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:47,618 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:47,618 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:47,618 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:47,718 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:47,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:47,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:47,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:47,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:47,722 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:47,722 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:47,722 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:47,723 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:47,723 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:47,723 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:47,724 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,724 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:47,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:47,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:47,724 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,724 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:47,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:47,725 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:47,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:47,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:47,725 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:47,726 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:47,726 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:47,726 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:47,726 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:47,726 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:47,726 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:47,726 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:47,726 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:47,726 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:47,726 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. as already flushing 2023-04-09 09:55:47,726 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:47,726 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:47,726 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:47,726 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,727 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:47,727 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:47,728 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,728 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:47,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:47,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:47,728 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:47,728 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:47,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:47,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:47,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:47,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:47,730 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:47,730 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d16f1dc[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:47,730 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:47,730 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:47,731 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:47,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:47,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:47,731 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,731 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:47,731 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:47,731 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:47,731 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:47,736 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,736 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:47,736 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:47,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:47,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:47,736 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:47,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:47,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:47,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:47,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:47,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:47,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:47,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:47,738 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:47,738 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:47,738 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:47,738 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:47,738 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:47,740 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:47,740 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:47,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:47,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:47,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:47,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:47,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:47,740 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:47,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:47,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:47,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:47,741 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:47,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:47,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:47,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:47,743 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:47,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:47,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,745 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,745 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:47,745 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:47,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:47,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:47,745 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:47,745 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:47,745 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:47,745 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:47,745 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:47,745 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,745 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:47,746 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:47,746 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:47,746 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:47,746 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:47,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:47,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:47,793 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:47,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59302 deadline: 1681034207787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,793 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:47,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59328 deadline: 1681034207787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,846 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:47,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:47,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:47,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:47,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:47,852 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:47,852 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:47,852 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:47,853 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:47,853 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:47,853 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:47,854 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,854 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:47,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:47,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:47,855 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,855 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:47,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:47,855 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:47,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:47,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:47,856 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:47,857 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:47,857 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:47,857 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:47,857 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:47,857 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:47,857 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:47,857 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:47,857 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:47,857 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:47,857 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. as already flushing 2023-04-09 09:55:47,857 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:47,857 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:47,857 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:47,857 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,858 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:47,858 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:47,860 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:47,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:47,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:47,860 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:47,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:47,861 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:47,861 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:47,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:47,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:47,863 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:47,863 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d5ed73c[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:47,863 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:47,863 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:47,864 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:47,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:47,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:47,865 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:47,865 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,865 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:47,865 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:47,865 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:47,866 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:47,866 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,866 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:47,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:47,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:47,866 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:47,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:47,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:47,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:47,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:47,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:47,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:47,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:47,868 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:47,869 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:47,869 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:47,869 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:47,869 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:47,871 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:47,871 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:47,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:47,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:47,871 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:47,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:47,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:47,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:47,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:47,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:47,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:47,873 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:47,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:47,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:47,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:47,874 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:47,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:47,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,880 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:47,881 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:47,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:47,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:47,881 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,881 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:47,881 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:47,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:47,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:47,881 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:47,881 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,881 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:47,881 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:47,881 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:47,881 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:47,882 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:47,882 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:47,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:47,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:47,882 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/e0186f035c7a4bcd9acbf6acbd512497 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/e0186f035c7a4bcd9acbf6acbd512497 2023-04-09 09:55:47,882 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/3aadb574b7ea48a39e4a1bc72db41022 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/3aadb574b7ea48a39e4a1bc72db41022 2023-04-09 09:55:47,887 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 2764d7f8155e1e5b0d641eebe063fa00/A of 2764d7f8155e1e5b0d641eebe063fa00 into 3aadb574b7ea48a39e4a1bc72db41022(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:55:47,887 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 2764d7f8155e1e5b0d641eebe063fa00/B of 2764d7f8155e1e5b0d641eebe063fa00 into e0186f035c7a4bcd9acbf6acbd512497(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:55:47,888 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2764d7f8155e1e5b0d641eebe063fa00: 2023-04-09 09:55:47,888 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2764d7f8155e1e5b0d641eebe063fa00: 2023-04-09 09:55:47,888 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00., storeName=2764d7f8155e1e5b0d641eebe063fa00/A, priority=11, startTime=1681034147391; duration=0sec 2023-04-09 09:55:47,888 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00., storeName=2764d7f8155e1e5b0d641eebe063fa00/B, priority=11, startTime=1681034147391; duration=0sec 2023-04-09 09:55:47,888 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:55:47,888 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:55:47,888 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-09 09:55:47,889 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60924 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-09 09:55:47,889 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1912): 2764d7f8155e1e5b0d641eebe063fa00/C is initiating minor compaction (all files) 2023-04-09 09:55:47,889 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2764d7f8155e1e5b0d641eebe063fa00/C in TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:47,889 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/9c35607a98664a6bb5ad65b69d70ea8f, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/e8af8ded416944b0852dd00cdc61689b, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/647c66655efd4bd9a7e8d1be179d7293, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/b0ee312e0dad46909d8690187b0246b4, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/ddac8b65087f438c87b0ac46bf99c85c] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp, totalSize=59.5 K 2023-04-09 09:55:47,890 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 9c35607a98664a6bb5ad65b69d70ea8f, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1681034143916 2023-04-09 09:55:47,890 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting e8af8ded416944b0852dd00cdc61689b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=229, earliestPutTs=1681034144295 2023-04-09 09:55:47,891 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 647c66655efd4bd9a7e8d1be179d7293, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=244, earliestPutTs=1681034145432 2023-04-09 09:55:47,891 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting b0ee312e0dad46909d8690187b0246b4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=264, earliestPutTs=1681034145803 2023-04-09 09:55:47,891 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting ddac8b65087f438c87b0ac46bf99c85c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=276, earliestPutTs=1681034146940 2023-04-09 09:55:47,894 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=312 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/8483cd68b8b74843ab838e4c89682d13 2023-04-09 09:55:47,903 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2764d7f8155e1e5b0d641eebe063fa00#C#compaction#475 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:55:47,981 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:47,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:47,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:47,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:47,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:47,985 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:47,985 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:47,985 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:47,985 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:47,986 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:47,986 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:47,987 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:47,987 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:47,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:47,987 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,987 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:47,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:47,988 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:47,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:47,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:47,988 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:47,988 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:47,988 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:47,988 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:47,989 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:47,989 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:47,989 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:47,989 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:47,989 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:47,989 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:47,989 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. as already flushing 2023-04-09 09:55:47,989 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:47,989 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:47,989 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:47,989 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,989 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:47,989 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:47,991 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,991 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:47,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:47,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:47,991 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:47,991 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:47,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:47,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:47,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:47,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:47,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:47,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12e20149[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:47,993 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:47,993 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:47,995 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:47,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:47,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:47,996 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:47,996 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:47,996 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,996 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:47,996 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:47,998 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:47,998 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:47,998 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:47,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:47,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:47,998 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:47,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:47,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:48,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:48,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:48,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:48,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:48,003 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:48,003 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:48,003 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:48,003 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:48,003 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:48,003 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:48,004 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:48,004 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:48,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:48,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:48,004 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:48,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:48,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:48,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:48,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:48,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:48,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:48,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:48,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:48,006 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:48,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:48,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:48,007 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:48,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,010 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:48,010 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:48,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:48,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:48,010 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,010 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:48,010 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:48,010 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:48,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:48,010 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:48,010 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:48,010 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:48,010 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:48,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:48,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:48,011 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:48,011 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:48,100 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:48,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59302 deadline: 1681034208095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,100 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:48,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59328 deadline: 1681034208096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,111 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:48,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:48,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:48,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:48,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:48,114 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:48,114 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:48,114 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:48,114 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:48,114 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:48,114 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:48,116 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:48,116 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:48,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:48,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:48,116 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,116 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:48,117 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:48,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:48,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:48,117 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:48,120 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:48,120 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:48,120 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:48,120 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:48,120 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:48,121 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:48,121 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:48,122 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:48,122 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:48,122 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. as already flushing 2023-04-09 09:55:48,122 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:48,122 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:48,123 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:48,123 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,123 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:48,123 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:48,125 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:48,125 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:48,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:48,125 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:48,125 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:48,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:48,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:48,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:48,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:48,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:48,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74e10a3d[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:48,126 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:48,126 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:48,127 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:48,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:48,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:48,127 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:48,127 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:48,127 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:48,128 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,128 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:48,129 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,129 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:48,129 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:48,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:48,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:48,129 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:48,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:48,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:48,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:48,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:48,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:48,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:48,131 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:48,131 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:48,131 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:48,131 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:48,131 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:48,131 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:48,136 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:48,136 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:48,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:48,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:48,136 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:48,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:48,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:48,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:48,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:48,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:48,137 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:48,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:48,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:48,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:48,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:48,138 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:48,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:48,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,140 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,140 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:48,140 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:48,140 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,140 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:48,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:48,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:48,140 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:48,141 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:48,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:48,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:48,141 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:48,141 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:48,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:48,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:48,140 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:48,141 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:48,141 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:48,241 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:48,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:48,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:48,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:48,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:48,248 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:48,248 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:48,248 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:48,248 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:48,249 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:48,249 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:48,250 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:48,250 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:48,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:48,250 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,250 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:48,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:48,251 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:48,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:48,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:48,251 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:48,251 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:48,251 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:48,251 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:48,251 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:48,251 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:48,252 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:48,252 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:48,252 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:48,252 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:48,252 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. as already flushing 2023-04-09 09:55:48,252 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:48,252 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:48,252 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:48,252 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,252 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:48,252 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:48,253 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:48,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:48,253 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:48,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:48,254 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:48,254 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:48,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:48,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:48,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:48,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:48,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e939d6c[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:48,255 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:48,255 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:48,262 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:48,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:48,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:48,263 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:48,263 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:48,263 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:48,263 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,263 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:48,264 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,264 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:48,265 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:48,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:48,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:48,265 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:48,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:48,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:48,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:48,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:48,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:48,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:48,268 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:48,268 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:48,268 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:48,268 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:48,268 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:48,268 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:48,270 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:48,270 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:48,270 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:48,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:48,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:48,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:48,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:48,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:48,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:48,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:48,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:48,272 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:48,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:48,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:48,273 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:48,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:48,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:48,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,277 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:48,277 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:48,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:48,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:48,277 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:48,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:48,277 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:48,278 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:48,278 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:48,278 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:48,278 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:48,278 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:48,278 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,278 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:48,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:48,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:48,278 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:48,278 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:48,306 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=312 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/872e331b74244e05b5d3fc4fce3d7a49 2023-04-09 09:55:48,316 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=312 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/22d7eb899f5f4d64a43b805192d09e8f 2023-04-09 09:55:48,320 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/a485f890737f489baf78df2141d90890 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/a485f890737f489baf78df2141d90890 2023-04-09 09:55:48,320 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/8483cd68b8b74843ab838e4c89682d13 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/8483cd68b8b74843ab838e4c89682d13 2023-04-09 09:55:48,323 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/8483cd68b8b74843ab838e4c89682d13, entries=200, sequenceid=312, filesize=14.3 K 2023-04-09 09:55:48,324 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 2764d7f8155e1e5b0d641eebe063fa00/C of 2764d7f8155e1e5b0d641eebe063fa00 into a485f890737f489baf78df2141d90890(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:55:48,324 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2764d7f8155e1e5b0d641eebe063fa00: 2023-04-09 09:55:48,324 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00., storeName=2764d7f8155e1e5b0d641eebe063fa00/C, priority=11, startTime=1681034147391; duration=0sec 2023-04-09 09:55:48,324 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:55:48,324 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/872e331b74244e05b5d3fc4fce3d7a49 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/872e331b74244e05b5d3fc4fce3d7a49 2023-04-09 09:55:48,327 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/872e331b74244e05b5d3fc4fce3d7a49, entries=150, sequenceid=312, filesize=11.9 K 2023-04-09 09:55:48,328 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/22d7eb899f5f4d64a43b805192d09e8f as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/22d7eb899f5f4d64a43b805192d09e8f 2023-04-09 09:55:48,330 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/22d7eb899f5f4d64a43b805192d09e8f, entries=150, sequenceid=312, filesize=11.9 K 2023-04-09 09:55:48,331 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 2764d7f8155e1e5b0d641eebe063fa00 in 902ms, sequenceid=312, compaction requested=false 2023-04-09 09:55:48,331 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2764d7f8155e1e5b0d641eebe063fa00: 2023-04-09 09:55:48,378 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:48,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:48,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:48,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:48,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:48,381 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:48,381 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:48,381 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:48,381 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:48,381 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:48,382 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:48,385 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,385 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:48,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:48,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:48,385 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,385 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:48,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:48,385 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:48,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:48,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:48,386 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:48,386 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:48,386 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:48,386 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:48,386 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:48,386 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:48,386 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:48,386 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:48,386 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:48,387 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:48,387 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2764d7f8155e1e5b0d641eebe063fa00 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-09 09:55:48,387 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=A 2023-04-09 09:55:48,387 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:48,387 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=B 2023-04-09 09:55:48,387 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:48,387 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=C 2023-04-09 09:55:48,387 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:48,394 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=325 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/d0174daf06b34a5e8aa2d734b18c5fe4 2023-04-09 09:55:48,403 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=325 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/f650f6c86ed945a9967d783281d7bbc9 2023-04-09 09:55:48,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 2764d7f8155e1e5b0d641eebe063fa00 2023-04-09 09:55:48,467 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. as already flushing 2023-04-09 09:55:48,506 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681034148493, max timestamp=1681034148504Num uniques 859; 2023-04-09 09:55:48,506 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681034148493, max timestamp=1681034148504Num uniques 859; 2023-04-09 09:55:48,506 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681034148493, max timestamp=1681034148504Num uniques 859; 2023-04-09 09:55:48,506 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-09 09:55:48,506 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-09 09:55:48,506 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-09 09:55:48,540 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:48,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59348 deadline: 1681034208531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,547 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:48,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59316 deadline: 1681034208539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,547 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:48,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59344 deadline: 1681034208540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,613 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:48,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59328 deadline: 1681034208604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,615 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:48,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59302 deadline: 1681034208610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,645 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:48,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59348 deadline: 1681034208641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,652 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:48,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59316 deadline: 1681034208648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,652 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:48,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59344 deadline: 1681034208648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,811 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=325 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/ddae70465267481b912c19d13ebf9477 2023-04-09 09:55:48,815 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/d0174daf06b34a5e8aa2d734b18c5fe4 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/d0174daf06b34a5e8aa2d734b18c5fe4 2023-04-09 09:55:48,818 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/d0174daf06b34a5e8aa2d734b18c5fe4, entries=150, sequenceid=325, filesize=11.9 K 2023-04-09 09:55:48,819 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/f650f6c86ed945a9967d783281d7bbc9 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/f650f6c86ed945a9967d783281d7bbc9 2023-04-09 09:55:48,822 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/f650f6c86ed945a9967d783281d7bbc9, entries=150, sequenceid=325, filesize=11.9 K 2023-04-09 09:55:48,823 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/ddae70465267481b912c19d13ebf9477 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/ddae70465267481b912c19d13ebf9477 2023-04-09 09:55:48,826 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/ddae70465267481b912c19d13ebf9477, entries=150, sequenceid=325, filesize=11.9 K 2023-04-09 09:55:48,826 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 2764d7f8155e1e5b0d641eebe063fa00 in 439ms, sequenceid=325, compaction requested=true 2023-04-09 09:55:48,826 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2764d7f8155e1e5b0d641eebe063fa00: 2023-04-09 09:55:48,826 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:48,826 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:48,826 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:48,826 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,826 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:48,826 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:48,828 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:48,828 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:48,829 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:48,829 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:48,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:48,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:48,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:48,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:48,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:48,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:48,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35df4eb5[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:48,832 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:48,832 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:48,833 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:48,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:48,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:48,833 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:48,833 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:48,833 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:48,833 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,833 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:48,834 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:48,834 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:48,834 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,834 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:48,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:48,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:48,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:48,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:48,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:48,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:48,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:48,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:48,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:48,837 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:48,837 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:48,837 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:48,837 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:48,837 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:48,839 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:48,839 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:48,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:48,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:48,839 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:48,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:48,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:48,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:48,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:48,840 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:48,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:48,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:48,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:48,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:48,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:48,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:48,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:48,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,842 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,844 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,844 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:48,844 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:48,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:48,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:48,844 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:48,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:48,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:48,844 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:48,844 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:48,844 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:48,844 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:48,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:48,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:48,844 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,845 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:48,845 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:48,845 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:48,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 2764d7f8155e1e5b0d641eebe063fa00 2023-04-09 09:55:48,882 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2764d7f8155e1e5b0d641eebe063fa00 3/3 column families, dataSize=154.31 KB heapSize=440.77 KB 2023-04-09 09:55:48,882 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=A 2023-04-09 09:55:48,882 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-09 09:55:48,882 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=B 2023-04-09 09:55:48,883 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-09 09:55:48,883 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=C 2023-04-09 09:55:48,883 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-09 09:55:48,890 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=364 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/cdebebce81934e9fa43b2474215ae2ca 2023-04-09 09:55:48,898 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=364 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/ccb64889195f4c928785ca9f959f5ff0 2023-04-09 09:55:48,899 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:48,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59344 deadline: 1681034208891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,899 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:48,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59316 deadline: 1681034208892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,902 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:48,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59348 deadline: 1681034208897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,908 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=364 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/466013a3c4f840f6bda2bdb2094a39f3 2023-04-09 09:55:48,911 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/cdebebce81934e9fa43b2474215ae2ca as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/cdebebce81934e9fa43b2474215ae2ca 2023-04-09 09:55:48,914 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/cdebebce81934e9fa43b2474215ae2ca, entries=150, sequenceid=364, filesize=11.9 K 2023-04-09 09:55:48,914 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/ccb64889195f4c928785ca9f959f5ff0 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/ccb64889195f4c928785ca9f959f5ff0 2023-04-09 09:55:48,921 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/ccb64889195f4c928785ca9f959f5ff0, entries=150, sequenceid=364, filesize=11.9 K 2023-04-09 09:55:48,922 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/466013a3c4f840f6bda2bdb2094a39f3 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/466013a3c4f840f6bda2bdb2094a39f3 2023-04-09 09:55:48,924 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/466013a3c4f840f6bda2bdb2094a39f3, entries=150, sequenceid=364, filesize=11.9 K 2023-04-09 09:55:48,925 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~440.72 KB/451296, currentSize=26.84 KB/27480 for 2764d7f8155e1e5b0d641eebe063fa00 in 43ms, sequenceid=364, compaction requested=true 2023-04-09 09:55:48,925 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2764d7f8155e1e5b0d641eebe063fa00: 2023-04-09 09:55:48,925 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:55:48,925 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:55:48,925 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:55:48,925 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-09 09:55:48,925 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-09 09:55:48,926 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49358 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-09 09:55:48,926 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51798 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-09 09:55:48,926 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): 2764d7f8155e1e5b0d641eebe063fa00/B is initiating minor compaction (all files) 2023-04-09 09:55:48,926 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1912): 2764d7f8155e1e5b0d641eebe063fa00/A is initiating minor compaction (all files) 2023-04-09 09:55:48,926 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2764d7f8155e1e5b0d641eebe063fa00/B in TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:48,926 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2764d7f8155e1e5b0d641eebe063fa00/A in TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:48,926 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/e0186f035c7a4bcd9acbf6acbd512497, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/872e331b74244e05b5d3fc4fce3d7a49, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/f650f6c86ed945a9967d783281d7bbc9, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/ccb64889195f4c928785ca9f959f5ff0] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp, totalSize=48.2 K 2023-04-09 09:55:48,927 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/3aadb574b7ea48a39e4a1bc72db41022, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/8483cd68b8b74843ab838e4c89682d13, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/d0174daf06b34a5e8aa2d734b18c5fe4, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/cdebebce81934e9fa43b2474215ae2ca] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp, totalSize=50.6 K 2023-04-09 09:55:48,927 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting e0186f035c7a4bcd9acbf6acbd512497, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=276, earliestPutTs=1681034146940 2023-04-09 09:55:48,927 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 3aadb574b7ea48a39e4a1bc72db41022, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=276, earliestPutTs=1681034146940 2023-04-09 09:55:48,927 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 872e331b74244e05b5d3fc4fce3d7a49, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1681034147094 2023-04-09 09:55:48,927 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 8483cd68b8b74843ab838e4c89682d13, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1681034147094 2023-04-09 09:55:48,927 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting f650f6c86ed945a9967d783281d7bbc9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=325, earliestPutTs=1681034147454 2023-04-09 09:55:48,927 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting d0174daf06b34a5e8aa2d734b18c5fe4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=325, earliestPutTs=1681034147454 2023-04-09 09:55:48,928 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting ccb64889195f4c928785ca9f959f5ff0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=364, earliestPutTs=1681034148851 2023-04-09 09:55:48,928 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting cdebebce81934e9fa43b2474215ae2ca, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=364, earliestPutTs=1681034148851 2023-04-09 09:55:48,934 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): 2764d7f8155e1e5b0d641eebe063fa00#B#compaction#483 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:55:48,934 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2764d7f8155e1e5b0d641eebe063fa00#A#compaction#484 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:55:48,944 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:48,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:48,946 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/9b8c16c693d845568cd38c0e9198c261 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/9b8c16c693d845568cd38c0e9198c261 2023-04-09 09:55:48,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:48,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:48,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:48,949 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:48,949 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:48,949 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:48,949 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:48,949 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:48,949 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:48,950 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:48,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:48,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:48,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:48,951 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2764d7f8155e1e5b0d641eebe063fa00/B of 2764d7f8155e1e5b0d641eebe063fa00 into 9b8c16c693d845568cd38c0e9198c261(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:55:48,951 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2764d7f8155e1e5b0d641eebe063fa00: 2023-04-09 09:55:48,951 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00., storeName=2764d7f8155e1e5b0d641eebe063fa00/B, priority=12, startTime=1681034148925; duration=0sec 2023-04-09 09:55:48,951 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:48,951 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:55:48,951 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-09 09:55:48,951 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:48,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:48,952 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:48,952 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:48,952 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:48,952 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:48,952 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:48,952 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:48,952 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:48,952 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:48,952 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:48,952 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:48,952 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:48,952 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:48,952 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2764d7f8155e1e5b0d641eebe063fa00 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-09 09:55:48,953 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49358 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-09 09:55:48,953 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): 2764d7f8155e1e5b0d641eebe063fa00/C is initiating minor compaction (all files) 2023-04-09 09:55:48,953 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2764d7f8155e1e5b0d641eebe063fa00/C in TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:48,953 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=A 2023-04-09 09:55:48,953 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/a485f890737f489baf78df2141d90890, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/22d7eb899f5f4d64a43b805192d09e8f, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/ddae70465267481b912c19d13ebf9477, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/466013a3c4f840f6bda2bdb2094a39f3] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp, totalSize=48.2 K 2023-04-09 09:55:48,953 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:48,953 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=B 2023-04-09 09:55:48,953 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:48,953 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=C 2023-04-09 09:55:48,953 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:48,953 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting a485f890737f489baf78df2141d90890, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=276, earliestPutTs=1681034146940 2023-04-09 09:55:48,954 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 22d7eb899f5f4d64a43b805192d09e8f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1681034147094 2023-04-09 09:55:48,954 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting ddae70465267481b912c19d13ebf9477, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=325, earliestPutTs=1681034147454 2023-04-09 09:55:48,955 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 466013a3c4f840f6bda2bdb2094a39f3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=364, earliestPutTs=1681034148851 2023-04-09 09:55:48,964 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=372 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/c3d37b6d3e1046c186779fc4e3c11ff5 2023-04-09 09:55:48,965 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): 2764d7f8155e1e5b0d641eebe063fa00#C#compaction#486 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:55:48,981 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/841654606d954311bc940b34ad580abc as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/841654606d954311bc940b34ad580abc 2023-04-09 09:55:48,985 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2764d7f8155e1e5b0d641eebe063fa00/C of 2764d7f8155e1e5b0d641eebe063fa00 into 841654606d954311bc940b34ad580abc(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:55:48,985 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2764d7f8155e1e5b0d641eebe063fa00: 2023-04-09 09:55:48,985 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00., storeName=2764d7f8155e1e5b0d641eebe063fa00/C, priority=12, startTime=1681034148925; duration=0sec 2023-04-09 09:55:48,985 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:55:49,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 2764d7f8155e1e5b0d641eebe063fa00 2023-04-09 09:55:49,004 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. as already flushing 2023-04-09 09:55:49,039 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681034149003, max timestamp=1681034149038Num uniques 859; 2023-04-09 09:55:49,039 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-09 09:55:49,040 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681034149003, max timestamp=1681034149038Num uniques 859; 2023-04-09 09:55:49,040 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681034149003, max timestamp=1681034149038Num uniques 859; 2023-04-09 09:55:49,040 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-09 09:55:49,040 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-09 09:55:49,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681034149063, max timestamp=1681034149073Num uniques 1068; 2023-04-09 09:55:49,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-09 09:55:49,078 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681034149063, max timestamp=1681034149073Num uniques 1068; 2023-04-09 09:55:49,078 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681034149063, max timestamp=1681034149073Num uniques 1068; 2023-04-09 09:55:49,078 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-09 09:55:49,078 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-09 09:55:49,128 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1681034149087, max timestamp=1681034149125Num uniques 1027; 2023-04-09 09:55:49,128 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1681034149087, max timestamp=1681034149125Num uniques 1027; 2023-04-09 09:55:49,128 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=12682, suffix heap size=134776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=277 2023-04-09 09:55:49,128 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=12682, suffix heap size=134776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=277 2023-04-09 09:55:49,128 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1681034149087, max timestamp=1681034149125Num uniques 1027; 2023-04-09 09:55:49,128 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=12682, suffix heap size=134776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=277 2023-04-09 09:55:49,172 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:49,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59344 deadline: 1681034209163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:49,174 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681034149153, max timestamp=1681034149172Num uniques 1136; 2023-04-09 09:55:49,174 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681034149153, max timestamp=1681034149172Num uniques 1136; 2023-04-09 09:55:49,174 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=10806, suffix heap size=146112,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=236 2023-04-09 09:55:49,174 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=10806, suffix heap size=146112,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=236 2023-04-09 09:55:49,174 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681034149153, max timestamp=1681034149172Num uniques 1136; 2023-04-09 09:55:49,174 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=10806, suffix heap size=146112,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=236 2023-04-09 09:55:49,230 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681034149213, max timestamp=1681034149227Num uniques 1095; 2023-04-09 09:55:49,230 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681034149213, max timestamp=1681034149227Num uniques 1095; 2023-04-09 09:55:49,230 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-09 09:55:49,230 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681034149213, max timestamp=1681034149227Num uniques 1095; 2023-04-09 09:55:49,230 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-09 09:55:49,230 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-09 09:55:49,271 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681034149239, max timestamp=1681034149269Num uniques 1054; 2023-04-09 09:55:49,271 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-09 09:55:49,271 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681034149239, max timestamp=1681034149269Num uniques 1054; 2023-04-09 09:55:49,271 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-09 09:55:49,271 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681034149239, max timestamp=1681034149269Num uniques 1054; 2023-04-09 09:55:49,271 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-09 09:55:49,309 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1681034149294, max timestamp=1681034149307Num uniques 1013; 2023-04-09 09:55:49,309 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1681034149294, max timestamp=1681034149307Num uniques 1013; 2023-04-09 09:55:49,309 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-09 09:55:49,309 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-09 09:55:49,309 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1681034149294, max timestamp=1681034149307Num uniques 1013; 2023-04-09 09:55:49,309 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-09 09:55:49,347 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/20a34cc1f59d4f27856b60a0687c4d13 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/20a34cc1f59d4f27856b60a0687c4d13 2023-04-09 09:55:49,347 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1681034149321, max timestamp=1681034149345Num uniques 1022; 2023-04-09 09:55:49,348 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=12452, suffix heap size=134256,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=272 2023-04-09 09:55:49,348 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1681034149321, max timestamp=1681034149345Num uniques 1022; 2023-04-09 09:55:49,348 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=12452, suffix heap size=134256,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=272 2023-04-09 09:55:49,348 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1681034149321, max timestamp=1681034149345Num uniques 1022; 2023-04-09 09:55:49,348 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=12452, suffix heap size=134256,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=272 2023-04-09 09:55:49,351 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2764d7f8155e1e5b0d641eebe063fa00/A of 2764d7f8155e1e5b0d641eebe063fa00 into 20a34cc1f59d4f27856b60a0687c4d13(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:55:49,351 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2764d7f8155e1e5b0d641eebe063fa00: 2023-04-09 09:55:49,351 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00., storeName=2764d7f8155e1e5b0d641eebe063fa00/A, priority=12, startTime=1681034148925; duration=0sec 2023-04-09 09:55:49,351 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:55:49,378 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=372 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/80bd0acffb5444c4a8e89a475dbc64b1 2023-04-09 09:55:49,394 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1681034149374, max timestamp=1681034149391Num uniques 1131; 2023-04-09 09:55:49,394 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=51796, new segment data size=10576, suffix heap size=145592,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1131, new segment cells count=231 2023-04-09 09:55:49,394 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1681034149374, max timestamp=1681034149391Num uniques 1131; 2023-04-09 09:55:49,394 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1681034149374, max timestamp=1681034149391Num uniques 1131; 2023-04-09 09:55:49,395 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=51796, new segment data size=10576, suffix heap size=145592,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1131, new segment cells count=231 2023-04-09 09:55:49,395 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=51796, new segment data size=10576, suffix heap size=145592,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1131, new segment cells count=231 2023-04-09 09:55:49,433 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1681034149419, max timestamp=1681034149431Num uniques 1090; 2023-04-09 09:55:49,433 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=13280, suffix heap size=141328,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=290 2023-04-09 09:55:49,434 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1681034149419, max timestamp=1681034149431Num uniques 1090; 2023-04-09 09:55:49,434 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1681034149420, max timestamp=1681034149431Num uniques 1090; 2023-04-09 09:55:49,434 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=10990, suffix heap size=141328,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=240 2023-04-09 09:55:49,434 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=13280, suffix heap size=141328,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=290 2023-04-09 09:55:49,464 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1681034149448, max timestamp=1681034149462Num uniques 1149; 2023-04-09 09:55:49,464 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1681034149448, max timestamp=1681034149462Num uniques 1149; 2023-04-09 09:55:49,464 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=13694, suffix heap size=147464,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=299 2023-04-09 09:55:49,464 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1681034149448, max timestamp=1681034149462Num uniques 1099; 2023-04-09 09:55:49,464 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=13694, suffix heap size=147464,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=299 2023-04-09 09:55:49,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=13694, suffix heap size=142264,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=299 2023-04-09 09:55:49,523 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:49,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59348 deadline: 1681034209519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:49,529 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:49,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59316 deadline: 1681034209520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:49,529 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:49,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59344 deadline: 1681034209521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:49,622 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:49,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59302 deadline: 1681034209621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:49,623 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:49,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59328 deadline: 1681034209622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:49,628 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:49,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59348 deadline: 1681034209624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:49,633 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:49,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59316 deadline: 1681034209630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:49,633 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:49,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59344 deadline: 1681034209630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:49,789 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=372 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/3660b8cde2914331988e32c2c52e3bac 2023-04-09 09:55:49,801 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/c3d37b6d3e1046c186779fc4e3c11ff5 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/c3d37b6d3e1046c186779fc4e3c11ff5 2023-04-09 09:55:49,806 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/c3d37b6d3e1046c186779fc4e3c11ff5, entries=100, sequenceid=372, filesize=9.5 K 2023-04-09 09:55:49,807 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/80bd0acffb5444c4a8e89a475dbc64b1 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/80bd0acffb5444c4a8e89a475dbc64b1 2023-04-09 09:55:49,814 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/80bd0acffb5444c4a8e89a475dbc64b1, entries=100, sequenceid=372, filesize=9.5 K 2023-04-09 09:55:49,815 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/3660b8cde2914331988e32c2c52e3bac as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/3660b8cde2914331988e32c2c52e3bac 2023-04-09 09:55:49,830 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/3660b8cde2914331988e32c2c52e3bac, entries=100, sequenceid=372, filesize=9.5 K 2023-04-09 09:55:49,831 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=154.31 KB/158010 for 2764d7f8155e1e5b0d641eebe063fa00 in 879ms, sequenceid=372, compaction requested=false 2023-04-09 09:55:49,831 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2764d7f8155e1e5b0d641eebe063fa00: 2023-04-09 09:55:49,831 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:49,831 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:49,832 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:49,832 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:49,832 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:49,832 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:49,834 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:49,834 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:49,834 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:49,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 2764d7f8155e1e5b0d641eebe063fa00 2023-04-09 09:55:49,837 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1681034149518, max timestamp=1681034149833Num uniques 1158; 2023-04-09 09:55:49,837 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1681034149518, max timestamp=1681034149833Num uniques 1158; 2023-04-09 09:55:49,837 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1681034149518, max timestamp=1681034149833Num uniques 1158; 2023-04-09 09:55:49,837 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=53030, new segment data size=7230, suffix heap size=148400,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1158, new segment cells count=158 2023-04-09 09:55:49,837 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2764d7f8155e1e5b0d641eebe063fa00 3/3 column families, dataSize=26.84 KB heapSize=66.33 KB 2023-04-09 09:55:49,837 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=53030, new segment data size=7230, suffix heap size=148400,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1158, new segment cells count=158 2023-04-09 09:55:49,837 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=53030, new segment data size=7230, suffix heap size=148400,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1158, new segment cells count=158 2023-04-09 09:55:49,837 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=A 2023-04-09 09:55:49,837 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:55:49,837 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=B 2023-04-09 09:55:49,837 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:55:49,837 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=C 2023-04-09 09:55:49,837 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:55:49,839 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:49,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:49,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:49,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:49,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:49,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:49,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:49,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:49,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:49,843 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:49,843 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3735f90e[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:49,843 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:49,843 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:49,845 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:49,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:49,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:49,846 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:49,846 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:49,846 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:49,846 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:49,846 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:49,848 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:49,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:49,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:49,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:49,849 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:49,849 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:49,849 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:49,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:49,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:49,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:49,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:49,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:49,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:49,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:49,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:49,854 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:49,854 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:49,854 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:49,854 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:49,854 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:49,854 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:49,855 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:49,855 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:49,855 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:49,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:49,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:49,856 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:49,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:49,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:49,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:49,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:49,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:49,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:49,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:49,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:49,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:49,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:49,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:49,858 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:49,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:49,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:49,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:49,861 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=584 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/e251a28a28744c259915b1dd81bc6adc 2023-04-09 09:55:49,864 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:49,864 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:49,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:49,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:49,865 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:49,865 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:49,865 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:49,865 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:49,865 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:49,865 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:49,865 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:49,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:49,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:49,865 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:49,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:49,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:49,865 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:49,865 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:49,866 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:49,880 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=584 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/7f7c64de611343a98691eb415352ff32 2023-04-09 09:55:49,889 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681034149868, max timestamp=1681034149887Num uniques 859; 2023-04-09 09:55:49,889 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681034149868, max timestamp=1681034149887Num uniques 859; 2023-04-09 09:55:49,889 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681034149868, max timestamp=1681034149887Num uniques 859; 2023-04-09 09:55:49,889 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-09 09:55:49,889 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-09 09:55:49,889 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-09 09:55:49,909 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=584 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/0f92c62541bc46e29c0d15560397f27c 2023-04-09 09:55:49,913 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/e251a28a28744c259915b1dd81bc6adc as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/e251a28a28744c259915b1dd81bc6adc 2023-04-09 09:55:49,916 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/e251a28a28744c259915b1dd81bc6adc, entries=150, sequenceid=584, filesize=11.9 K 2023-04-09 09:55:49,917 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/7f7c64de611343a98691eb415352ff32 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/7f7c64de611343a98691eb415352ff32 2023-04-09 09:55:49,920 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/7f7c64de611343a98691eb415352ff32, entries=150, sequenceid=584, filesize=11.9 K 2023-04-09 09:55:49,921 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/0f92c62541bc46e29c0d15560397f27c as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/0f92c62541bc46e29c0d15560397f27c 2023-04-09 09:55:49,923 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/0f92c62541bc46e29c0d15560397f27c, entries=150, sequenceid=584, filesize=11.9 K 2023-04-09 09:55:49,924 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~66.28 KB/67872, currentSize=120.76 KB/123660 for 2764d7f8155e1e5b0d641eebe063fa00 in 87ms, sequenceid=584, compaction requested=true 2023-04-09 09:55:49,924 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2764d7f8155e1e5b0d641eebe063fa00: 2023-04-09 09:55:49,924 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:55:49,925 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-09 09:55:49,925 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-09 09:55:49,925 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:55:49,925 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:55:49,925 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34864 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-09 09:55:49,925 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34864 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-09 09:55:49,925 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1912): 2764d7f8155e1e5b0d641eebe063fa00/A is initiating minor compaction (all files) 2023-04-09 09:55:49,926 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): 2764d7f8155e1e5b0d641eebe063fa00/B is initiating minor compaction (all files) 2023-04-09 09:55:49,926 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2764d7f8155e1e5b0d641eebe063fa00/A in TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:49,926 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2764d7f8155e1e5b0d641eebe063fa00/B in TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:49,926 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/20a34cc1f59d4f27856b60a0687c4d13, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/c3d37b6d3e1046c186779fc4e3c11ff5, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/e251a28a28744c259915b1dd81bc6adc] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp, totalSize=34.0 K 2023-04-09 09:55:49,926 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/9b8c16c693d845568cd38c0e9198c261, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/80bd0acffb5444c4a8e89a475dbc64b1, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/7f7c64de611343a98691eb415352ff32] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp, totalSize=34.0 K 2023-04-09 09:55:49,926 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 20a34cc1f59d4f27856b60a0687c4d13, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=364, earliestPutTs=1681034148851 2023-04-09 09:55:49,926 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 9b8c16c693d845568cd38c0e9198c261, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=364, earliestPutTs=1681034148851 2023-04-09 09:55:49,926 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting c3d37b6d3e1046c186779fc4e3c11ff5, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1681034148891 2023-04-09 09:55:49,926 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 80bd0acffb5444c4a8e89a475dbc64b1, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1681034148891 2023-04-09 09:55:49,927 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting e251a28a28744c259915b1dd81bc6adc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=584, earliestPutTs=1681034149518 2023-04-09 09:55:49,927 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 7f7c64de611343a98691eb415352ff32, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=584, earliestPutTs=1681034149518 2023-04-09 09:55:49,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 2764d7f8155e1e5b0d641eebe063fa00 2023-04-09 09:55:49,929 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2764d7f8155e1e5b0d641eebe063fa00 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-09 09:55:49,929 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=A 2023-04-09 09:55:49,929 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:55:49,929 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=B 2023-04-09 09:55:49,929 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:55:49,929 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=C 2023-04-09 09:55:49,930 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:55:49,935 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2764d7f8155e1e5b0d641eebe063fa00#A#compaction#493 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:55:49,936 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): 2764d7f8155e1e5b0d641eebe063fa00#B#compaction#494 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:55:49,948 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:49,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59344 deadline: 1681034209942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:49,948 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:49,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59348 deadline: 1681034209943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:49,951 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:49,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59316 deadline: 1681034209947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:49,955 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=619 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/230a11401f0441d29e6c8454dbba3b8c 2023-04-09 09:55:49,965 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:49,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:49,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:49,970 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/e6f4fdd054444fd7a55b8cd7a41de80b as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/e6f4fdd054444fd7a55b8cd7a41de80b 2023-04-09 09:55:49,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:49,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:49,971 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:49,971 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:49,971 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:49,971 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:49,972 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:49,972 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:49,973 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:49,973 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:49,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:49,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:49,973 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:49,973 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:49,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:49,974 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=619 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/897b5758893940f8ac979ebf516258b6 2023-04-09 09:55:49,974 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:49,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:49,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:49,974 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:49,975 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:49,975 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:49,975 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:49,975 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:49,975 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:49,975 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:49,975 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:49,975 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:49,976 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:49,976 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. as already flushing 2023-04-09 09:55:49,976 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:49,976 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:49,976 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:49,976 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:49,976 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:49,976 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:49,976 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2764d7f8155e1e5b0d641eebe063fa00/B of 2764d7f8155e1e5b0d641eebe063fa00 into e6f4fdd054444fd7a55b8cd7a41de80b(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:55:49,976 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2764d7f8155e1e5b0d641eebe063fa00: 2023-04-09 09:55:49,976 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00., storeName=2764d7f8155e1e5b0d641eebe063fa00/B, priority=13, startTime=1681034149925; duration=0sec 2023-04-09 09:55:49,976 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:55:49,976 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-09 09:55:49,977 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:49,977 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:49,977 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34864 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-09 09:55:49,978 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): 2764d7f8155e1e5b0d641eebe063fa00/C is initiating minor compaction (all files) 2023-04-09 09:55:49,978 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:49,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:49,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:49,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:49,978 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:49,978 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2764d7f8155e1e5b0d641eebe063fa00/C in TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:49,978 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/841654606d954311bc940b34ad580abc, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/3660b8cde2914331988e32c2c52e3bac, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/0f92c62541bc46e29c0d15560397f27c] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp, totalSize=34.0 K 2023-04-09 09:55:49,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:49,978 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 841654606d954311bc940b34ad580abc, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=364, earliestPutTs=1681034148851 2023-04-09 09:55:49,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:49,979 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 3660b8cde2914331988e32c2c52e3bac, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1681034148891 2023-04-09 09:55:49,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:49,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:49,979 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 0f92c62541bc46e29c0d15560397f27c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=584, earliestPutTs=1681034149518 2023-04-09 09:55:49,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:49,980 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:49,980 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b79bd37[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:49,980 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:49,980 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:49,982 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:49,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:49,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:49,982 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:49,982 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:49,982 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:49,982 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:49,982 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:49,985 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:49,985 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:49,985 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:49,985 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:49,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:49,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:49,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:49,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:49,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:49,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:49,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:49,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:49,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:49,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:49,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:49,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:49,990 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:49,990 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:49,990 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:49,990 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:49,990 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:49,991 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:49,991 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:49,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:49,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:49,991 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:49,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:49,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:49,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:49,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:49,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:49,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:49,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:49,994 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:49,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:49,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:49,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:49,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:49,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:49,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:49,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:49,998 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,004 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): 2764d7f8155e1e5b0d641eebe063fa00#C#compaction#497 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:55:50,005 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,005 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:50,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:50,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:50,005 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:50,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:50,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:50,005 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:50,005 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:50,005 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,005 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:50,005 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:50,006 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:50,005 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:50,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:50,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:50,005 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:50,006 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:50,010 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=619 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/dee7aecf53d94bfe8036780b006f6dcc 2023-04-09 09:55:50,015 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/230a11401f0441d29e6c8454dbba3b8c as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/230a11401f0441d29e6c8454dbba3b8c 2023-04-09 09:55:50,019 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/230a11401f0441d29e6c8454dbba3b8c, entries=150, sequenceid=619, filesize=11.9 K 2023-04-09 09:55:50,019 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/f8aae433e960426d9a7de0833e0d70d1 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/f8aae433e960426d9a7de0833e0d70d1 2023-04-09 09:55:50,019 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/897b5758893940f8ac979ebf516258b6 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/897b5758893940f8ac979ebf516258b6 2023-04-09 09:55:50,023 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2764d7f8155e1e5b0d641eebe063fa00/C of 2764d7f8155e1e5b0d641eebe063fa00 into f8aae433e960426d9a7de0833e0d70d1(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:55:50,023 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2764d7f8155e1e5b0d641eebe063fa00: 2023-04-09 09:55:50,023 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/897b5758893940f8ac979ebf516258b6, entries=150, sequenceid=619, filesize=11.9 K 2023-04-09 09:55:50,023 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00., storeName=2764d7f8155e1e5b0d641eebe063fa00/C, priority=13, startTime=1681034149925; duration=0sec 2023-04-09 09:55:50,023 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:55:50,024 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/dee7aecf53d94bfe8036780b006f6dcc as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/dee7aecf53d94bfe8036780b006f6dcc 2023-04-09 09:55:50,027 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/dee7aecf53d94bfe8036780b006f6dcc, entries=150, sequenceid=619, filesize=11.9 K 2023-04-09 09:55:50,029 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for 2764d7f8155e1e5b0d641eebe063fa00 in 100ms, sequenceid=619, compaction requested=false 2023-04-09 09:55:50,029 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2764d7f8155e1e5b0d641eebe063fa00: 2023-04-09 09:55:50,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 2764d7f8155e1e5b0d641eebe063fa00 2023-04-09 09:55:50,055 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2764d7f8155e1e5b0d641eebe063fa00 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-09 09:55:50,056 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=A 2023-04-09 09:55:50,056 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:50,056 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=B 2023-04-09 09:55:50,056 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:50,056 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=C 2023-04-09 09:55:50,056 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:50,066 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=635 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/7c9e3ced0699439fb67d36440eaef12c 2023-04-09 09:55:50,082 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=635 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/0c5bb1f9f9dc4bec95f6806bf3e56b5d 2023-04-09 09:55:50,092 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=635 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/87947492ff71469eb3f2521b02f79b2f 2023-04-09 09:55:50,096 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/7c9e3ced0699439fb67d36440eaef12c as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/7c9e3ced0699439fb67d36440eaef12c 2023-04-09 09:55:50,098 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/7c9e3ced0699439fb67d36440eaef12c, entries=300, sequenceid=635, filesize=19.0 K 2023-04-09 09:55:50,099 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/0c5bb1f9f9dc4bec95f6806bf3e56b5d as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/0c5bb1f9f9dc4bec95f6806bf3e56b5d 2023-04-09 09:55:50,101 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:50,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59344 deadline: 1681034210094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,103 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/0c5bb1f9f9dc4bec95f6806bf3e56b5d, entries=150, sequenceid=635, filesize=11.9 K 2023-04-09 09:55:50,103 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/87947492ff71469eb3f2521b02f79b2f as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/87947492ff71469eb3f2521b02f79b2f 2023-04-09 09:55:50,106 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:50,106 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/87947492ff71469eb3f2521b02f79b2f, entries=150, sequenceid=635, filesize=11.9 K 2023-04-09 09:55:50,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:50,107 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 2764d7f8155e1e5b0d641eebe063fa00 in 53ms, sequenceid=635, compaction requested=true 2023-04-09 09:55:50,107 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2764d7f8155e1e5b0d641eebe063fa00: 2023-04-09 09:55:50,107 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:55:50,107 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:55:50,107 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-09 09:55:50,107 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 3 compacting, 2 eligible, 16 blocking 2023-04-09 09:55:50,107 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-09 09:55:50,108 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-09 09:55:50,108 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. because compaction request was cancelled 2023-04-09 09:55:50,108 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-09 09:55:50,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:50,109 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-09 09:55:50,109 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): 2764d7f8155e1e5b0d641eebe063fa00/B is initiating minor compaction (all files) 2023-04-09 09:55:50,109 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2764d7f8155e1e5b0d641eebe063fa00/B in TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:50,109 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/e6f4fdd054444fd7a55b8cd7a41de80b, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/897b5758893940f8ac979ebf516258b6, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/0c5bb1f9f9dc4bec95f6806bf3e56b5d] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp, totalSize=36.5 K 2023-04-09 09:55:50,109 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting e6f4fdd054444fd7a55b8cd7a41de80b, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=584, earliestPutTs=1681034149518 2023-04-09 09:55:50,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 2764d7f8155e1e5b0d641eebe063fa00 2023-04-09 09:55:50,110 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2764d7f8155e1e5b0d641eebe063fa00 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-09 09:55:50,110 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 897b5758893940f8ac979ebf516258b6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=619, earliestPutTs=1681034149919 2023-04-09 09:55:50,110 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=A 2023-04-09 09:55:50,110 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:50,110 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=B 2023-04-09 09:55:50,110 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:50,110 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=C 2023-04-09 09:55:50,110 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:50,111 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 0c5bb1f9f9dc4bec95f6806bf3e56b5d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=635, earliestPutTs=1681034150052 2023-04-09 09:55:50,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:50,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:50,111 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:50,111 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:50,111 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:50,111 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:50,111 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:50,111 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:50,112 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,112 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:50,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:50,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:50,113 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,113 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:50,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:50,113 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:50,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:50,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:50,114 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:50,120 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:50,120 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:50,120 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:50,120 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:50,120 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:50,120 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:50,120 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:50,120 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:50,120 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:50,120 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. as already flushing 2023-04-09 09:55:50,121 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:50,121 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:50,121 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:50,121 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,121 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:50,121 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:50,121 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): 2764d7f8155e1e5b0d641eebe063fa00#B#compaction#502 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:55:50,123 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:50,123 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:50,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:50,123 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:50,123 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:50,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:50,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:50,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:50,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:50,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:50,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@498331c9[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:50,126 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:50,126 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:50,127 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:50,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:50,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:50,127 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:50,127 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:50,127 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:50,127 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,127 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:50,128 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:50,128 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,128 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:50,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:50,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:50,129 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:50,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:50,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:50,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:50,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,130 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/df01f61a1ec1429ea0b551f4665aca74 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/df01f61a1ec1429ea0b551f4665aca74 2023-04-09 09:55:50,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:50,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:50,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:50,131 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:50,131 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:50,131 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:50,131 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:50,131 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:50,131 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:50,133 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:50,133 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:50,133 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:50,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:50,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:50,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:50,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:50,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:50,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:50,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:50,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:50,134 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2764d7f8155e1e5b0d641eebe063fa00/B of 2764d7f8155e1e5b0d641eebe063fa00 into df01f61a1ec1429ea0b551f4665aca74(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:55:50,134 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2764d7f8155e1e5b0d641eebe063fa00: 2023-04-09 09:55:50,134 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00., storeName=2764d7f8155e1e5b0d641eebe063fa00/B, priority=13, startTime=1681034150107; duration=0sec 2023-04-09 09:55:50,134 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:55:50,134 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-09 09:55:50,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:50,135 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:50,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:50,135 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-09 09:55:50,135 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): 2764d7f8155e1e5b0d641eebe063fa00/C is initiating minor compaction (all files) 2023-04-09 09:55:50,136 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2764d7f8155e1e5b0d641eebe063fa00/C in TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:50,136 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/f8aae433e960426d9a7de0833e0d70d1, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/dee7aecf53d94bfe8036780b006f6dcc, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/87947492ff71469eb3f2521b02f79b2f] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp, totalSize=36.5 K 2023-04-09 09:55:50,136 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting f8aae433e960426d9a7de0833e0d70d1, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=584, earliestPutTs=1681034149518 2023-04-09 09:55:50,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:50,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,137 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:50,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:50,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,140 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting dee7aecf53d94bfe8036780b006f6dcc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=619, earliestPutTs=1681034149919 2023-04-09 09:55:50,140 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,140 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:50,140 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:50,140 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:50,140 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:50,140 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 87947492ff71469eb3f2521b02f79b2f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=635, earliestPutTs=1681034150052 2023-04-09 09:55:50,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:50,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:50,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:50,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:50,140 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:50,141 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,141 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:50,141 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:50,141 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:50,141 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:50,141 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:50,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:50,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:50,145 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): 2764d7f8155e1e5b0d641eebe063fa00#C#compaction#503 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:55:50,153 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:50,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59316 deadline: 1681034210144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,155 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:50,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59348 deadline: 1681034210150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,204 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:50,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59344 deadline: 1681034210202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,241 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:50,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:50,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:50,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:50,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:50,244 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:50,244 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:50,244 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:50,244 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:50,244 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:50,244 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:50,246 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:50,246 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:50,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:50,246 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,246 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:50,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:50,246 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:50,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:50,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:50,247 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:50,247 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:50,247 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:50,247 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:50,247 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:50,247 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:50,247 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:50,247 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:50,247 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:50,247 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:50,247 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. as already flushing 2023-04-09 09:55:50,247 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:50,247 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:50,248 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:50,248 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,248 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:50,248 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:50,249 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,249 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:50,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:50,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:50,249 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:50,249 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:50,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:50,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:50,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:50,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:50,251 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:50,251 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66664aa[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:50,251 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:50,251 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:50,252 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:50,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:50,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:50,252 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:50,252 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,252 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:50,252 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:50,252 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:50,254 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,254 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:50,254 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:50,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:50,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:50,255 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:50,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:50,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:50,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:50,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:50,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:50,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:50,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:50,257 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:50,257 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:50,257 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:50,257 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:50,257 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:50,258 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:50,258 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:50,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:50,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:50,258 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:50,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:50,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:50,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:50,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:50,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:50,259 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:50,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:50,259 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:50,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59316 deadline: 1681034210254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:50,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:50,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:50,260 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:50,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:50,261 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:50,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59348 deadline: 1681034210256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,262 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:50,263 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:50,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:50,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:50,263 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:50,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:50,263 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:50,263 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:50,263 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:50,263 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:50,263 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:50,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:50,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:50,263 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,263 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:50,263 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:50,263 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:50,263 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:50,363 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:50,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:50,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:50,367 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/b1e58e24eef54e16bc7b0a877bf2d790 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/b1e58e24eef54e16bc7b0a877bf2d790 2023-04-09 09:55:50,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:50,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:50,367 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:50,367 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:50,367 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:50,367 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:50,368 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:50,368 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:50,374 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:50,374 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:50,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:50,374 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,374 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:50,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:50,374 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2764d7f8155e1e5b0d641eebe063fa00/A of 2764d7f8155e1e5b0d641eebe063fa00 into b1e58e24eef54e16bc7b0a877bf2d790(size=12.7 K), total size for store is 43.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:55:50,375 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2764d7f8155e1e5b0d641eebe063fa00: 2023-04-09 09:55:50,375 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00., storeName=2764d7f8155e1e5b0d641eebe063fa00/A, priority=13, startTime=1681034149924; duration=0sec 2023-04-09 09:55:50,375 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:55:50,375 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:50,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:50,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:50,376 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:50,376 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:50,376 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:50,376 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:50,376 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:50,376 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:50,377 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:50,377 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:50,377 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:50,377 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:50,377 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. as already flushing 2023-04-09 09:55:50,377 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:50,378 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:50,378 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:50,378 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,378 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:50,378 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:50,380 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:50,380 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:50,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:50,380 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:50,382 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:50,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:50,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:50,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:50,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:50,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:50,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e8300e1[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:50,383 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:50,383 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:50,384 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:50,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:50,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:50,386 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:50,386 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:50,386 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:50,386 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,386 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:50,388 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:50,388 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:50,388 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:50,388 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:50,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:50,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:50,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:50,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:50,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:50,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:50,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:50,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:50,392 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:50,392 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:50,392 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:50,392 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:50,392 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:50,401 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:50,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:50,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:50,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:50,401 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:50,401 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:50,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:50,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:50,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:50,402 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:50,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:50,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:50,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:50,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:50,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:50,403 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:50,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:50,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,408 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,408 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:50,408 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:50,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:50,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:50,409 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:50,409 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,409 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:50,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:50,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:50,408 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:50,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:50,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:50,409 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:50,409 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:50,409 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:50,409 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:50,409 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:50,413 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:50,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59344 deadline: 1681034210407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,465 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:50,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59316 deadline: 1681034210461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,465 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:50,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59348 deadline: 1681034210463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,509 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:50,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:50,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:50,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:50,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:50,513 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:50,513 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:50,513 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:50,513 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:50,513 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:50,513 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:50,514 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,514 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:50,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:50,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:50,515 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,515 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:50,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:50,515 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:50,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:50,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:50,515 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:50,516 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:50,516 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:50,516 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:50,516 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:50,516 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:50,516 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:50,516 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:50,516 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:50,516 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:50,516 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. as already flushing 2023-04-09 09:55:50,516 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:50,516 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:50,516 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:50,516 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,517 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:50,517 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:50,518 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:50,518 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:50,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:50,518 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:50,518 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:50,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:50,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:50,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:50,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:50,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:50,520 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18958603[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:50,520 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:50,520 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:50,521 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:50,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:50,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:50,521 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:50,521 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:50,521 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,521 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:50,521 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:50,522 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=655 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/6567c7954953486e8877579f00b1bc34 2023-04-09 09:55:50,523 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,523 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:50,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:50,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:50,523 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:50,523 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:50,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:50,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:50,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:50,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:50,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:50,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:50,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:50,528 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:50,528 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:50,528 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:50,528 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:50,528 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:50,529 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:50,529 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:50,529 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:50,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:50,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:50,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:50,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:50,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:50,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:50,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:50,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:50,530 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:50,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:50,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:50,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:50,531 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:50,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:50,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,534 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,535 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:50,535 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:50,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:50,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:50,535 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:50,535 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:50,535 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:50,535 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,535 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:50,535 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:50,535 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:50,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:50,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:50,535 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:50,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:50,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:50,535 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:50,535 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:50,553 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/f45909998c0d4fc7b46e9ee0e347901d as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/f45909998c0d4fc7b46e9ee0e347901d 2023-04-09 09:55:50,556 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2764d7f8155e1e5b0d641eebe063fa00/C of 2764d7f8155e1e5b0d641eebe063fa00 into f45909998c0d4fc7b46e9ee0e347901d(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:55:50,556 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2764d7f8155e1e5b0d641eebe063fa00: 2023-04-09 09:55:50,557 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00., storeName=2764d7f8155e1e5b0d641eebe063fa00/C, priority=13, startTime=1681034150107; duration=0sec 2023-04-09 09:55:50,557 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:55:50,636 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:50,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:50,637 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:50,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:50,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:50,639 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:50,639 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:50,640 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:50,640 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:50,640 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:50,640 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:50,641 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:50,641 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:50,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:50,641 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,641 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:50,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:50,642 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:50,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:50,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:50,642 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:50,642 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:50,642 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:50,642 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:50,642 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:50,642 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:50,643 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:50,643 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:50,643 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:50,643 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:50,643 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. as already flushing 2023-04-09 09:55:50,643 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:50,643 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:50,643 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:50,643 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,643 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:50,643 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:50,645 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:50,645 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:50,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:50,645 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:50,645 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:50,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:50,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:50,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:50,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:50,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:50,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4848f1a1[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:50,647 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:50,647 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:50,648 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:50,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:50,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:50,648 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:50,648 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:50,648 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,648 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:50,648 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:50,649 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:50,649 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:50,649 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,649 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:50,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:50,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:50,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:50,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:50,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:50,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:50,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:50,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:50,651 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:50,651 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:50,651 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:50,651 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:50,651 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:50,651 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:50,653 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:50,653 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:50,653 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:50,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:50,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:50,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:50,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:50,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:50,653 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:50,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:50,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:50,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:50,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:50,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:50,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:50,654 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:50,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:50,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,657 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:50,657 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,657 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:50,657 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:50,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:50,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:50,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:50,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:50,657 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:50,657 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:50,658 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:50,658 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:50,658 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:50,658 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:50,658 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:50,658 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:50,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:50,658 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:50,717 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:50,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59344 deadline: 1681034210715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,758 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:50,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:50,760 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:50,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:50,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:50,761 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:50,761 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:50,761 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:50,761 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:50,762 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:50,762 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:50,763 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:50,763 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:50,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:50,763 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,763 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:50,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:50,763 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:50,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:50,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:50,764 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:50,764 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:50,764 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:50,764 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:50,764 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:50,764 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:50,764 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:50,764 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:50,764 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:50,764 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:50,765 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. as already flushing 2023-04-09 09:55:50,765 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:50,765 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:50,765 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:50,765 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,765 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:50,765 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:50,768 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,768 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:50,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:50,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:50,769 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:50,769 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:50,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:50,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:50,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:50,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:50,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:50,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7dbe9f36[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:50,770 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:50,770 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:50,772 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:50,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:50,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:50,772 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:50,772 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:50,772 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,772 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:50,772 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:50,773 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:50,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59316 deadline: 1681034210768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,773 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:50,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59348 deadline: 1681034210768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,773 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:50,773 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:50,773 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:50,774 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:50,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:50,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:50,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:50,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:50,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:50,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:50,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:50,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:50,776 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:50,777 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:50,777 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:50,777 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:50,777 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:50,778 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:50,778 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:50,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:50,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:50,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:50,778 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:50,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:50,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:50,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:50,778 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:50,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:50,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:50,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:50,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:50,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:50,780 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:50,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:50,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,783 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,783 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:50,783 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:50,783 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:50,783 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:50,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:50,783 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:50,783 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:50,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:50,783 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:50,783 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:50,783 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:50,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:50,783 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:50,784 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:50,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:50,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:50,884 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:50,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:50,886 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:50,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:50,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:50,887 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:50,887 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:50,887 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:50,887 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:50,887 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:50,887 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:50,889 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:50,889 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:50,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:50,889 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,889 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:50,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:50,890 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:50,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:50,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:50,890 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:50,890 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:50,890 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:50,890 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:50,890 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:50,890 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:50,890 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:50,890 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:50,890 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:50,890 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:50,891 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. as already flushing 2023-04-09 09:55:50,891 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:50,891 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:50,891 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:50,891 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,891 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:50,891 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:50,892 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,892 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:50,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:50,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:50,892 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:50,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:50,892 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:50,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:50,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:50,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:50,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:50,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7bbc1c68[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:50,894 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:50,894 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:50,895 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:50,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:50,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:50,895 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:50,895 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:50,895 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:50,895 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,895 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:50,896 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,896 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:50,896 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:50,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:50,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:50,896 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:50,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:50,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:50,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:50,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:50,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:50,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:50,898 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:50,898 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:50,898 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:50,898 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:50,898 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:50,898 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:50,901 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:50,901 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:50,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:50,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:50,901 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:50,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:50,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:50,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:50,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:50,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:50,901 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:50,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:50,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:50,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:50,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:50,902 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:50,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:50,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,904 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,904 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:50,904 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:50,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:50,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:50,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:50,904 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:50,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:50,904 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:50,904 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:50,904 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:50,904 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:50,904 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:50,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:50,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:50,905 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:50,905 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:50,905 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:50,905 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:50,933 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=655 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/b1d903ec662641b29368389476396ea4 2023-04-09 09:55:51,005 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:51,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:51,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:51,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:51,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:51,012 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:51,012 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:51,012 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:51,012 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:51,013 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:51,013 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:51,014 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:51,014 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:51,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:51,015 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,015 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:51,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:51,015 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:51,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:51,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:51,016 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:51,016 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:51,016 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:51,016 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:51,017 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:51,017 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:51,017 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:51,017 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:51,017 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:51,017 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:51,017 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. as already flushing 2023-04-09 09:55:51,017 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:51,017 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:51,017 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:51,017 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,017 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:51,017 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:51,018 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:51,018 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:51,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:51,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:51,019 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:51,019 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:51,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:51,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:51,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:51,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:51,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fad3882[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:51,020 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:51,020 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:51,022 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:51,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:51,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:51,022 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,022 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:51,022 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:51,023 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:51,023 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:51,024 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:51,024 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,024 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:51,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:51,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:51,024 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:51,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:51,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:51,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:51,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:51,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:51,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:51,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:51,026 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:51,026 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:51,026 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:51,026 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:51,027 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:51,028 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:51,028 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:51,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:51,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:51,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:51,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:51,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:51,028 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:51,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:51,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:51,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:51,029 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:51,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:51,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:51,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:51,030 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:51,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:51,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,033 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,033 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:51,033 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:51,033 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:51,033 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,033 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:51,034 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:51,034 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:51,034 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:51,034 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:51,034 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:51,034 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:51,034 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:51,035 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:51,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:51,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:51,035 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:51,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:51,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:51,134 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:51,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:51,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:51,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:51,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:51,137 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:51,138 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:51,138 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:51,138 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:51,138 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:51,138 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:51,140 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:51,140 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:51,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:51,140 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,140 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:51,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:51,141 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:51,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:51,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:51,141 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:51,141 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:51,141 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:51,141 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:51,141 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:51,142 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:51,142 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:51,142 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:51,142 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:51,142 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:51,142 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. as already flushing 2023-04-09 09:55:51,142 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:51,142 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:51,142 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:51,142 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,142 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:51,142 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:51,143 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,143 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:51,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:51,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:51,144 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:51,144 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:51,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:51,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:51,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:51,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:51,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:51,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40e70d2f[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:51,145 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:51,145 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:51,146 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:51,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:51,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:51,146 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:51,146 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:51,146 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:51,146 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,146 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:51,153 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,153 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:51,153 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:51,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:51,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:51,153 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:51,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:51,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:51,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:51,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:51,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:51,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:51,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:51,155 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:51,155 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:51,155 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:51,155 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:51,155 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:51,157 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:51,157 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:51,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:51,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:51,157 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:51,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:51,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:51,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:51,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:51,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:51,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:51,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:51,157 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:51,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:51,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:51,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,159 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:51,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:51,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,161 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,161 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:51,161 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:51,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:51,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:51,161 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:51,161 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:51,161 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:51,161 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:51,161 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:51,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:51,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:51,161 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:51,161 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:51,161 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,162 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:51,162 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:51,162 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:51,219 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:51,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59344 deadline: 1681034211218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,262 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:51,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:51,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:51,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:51,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:51,265 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:51,265 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:51,265 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:51,265 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:51,265 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:51,265 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:51,266 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:51,266 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:51,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:51,267 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:51,267 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:51,267 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:51,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:51,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:51,267 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:51,268 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:51,268 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:51,268 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:51,268 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:51,268 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:51,268 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:51,268 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:51,268 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:51,268 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:51,268 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. as already flushing 2023-04-09 09:55:51,268 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:51,268 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:51,268 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:51,268 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,269 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:51,269 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:51,271 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,271 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:51,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:51,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:51,271 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:51,271 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:51,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:51,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:51,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:51,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:51,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:51,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49297d2a[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:51,274 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:51,274 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:51,275 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:51,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:51,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:51,275 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:51,275 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:51,275 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:51,275 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,275 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:51,276 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:51,276 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:51,276 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,276 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:51,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:51,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:51,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:51,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:51,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:51,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:51,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:51,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:51,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:51,279 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:51,279 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:51,279 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:51,279 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:51,279 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:51,280 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:51,280 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:51,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:51,280 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:51,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:51,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:51,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:51,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:51,280 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:51,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59316 deadline: 1681034211276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,281 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:51,281 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:51,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59348 deadline: 1681034211277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:51,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:51,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:51,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:51,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:51,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:51,282 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:51,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:51,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,284 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,285 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:51,285 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:51,285 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:51,285 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:51,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:51,285 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:51,285 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:51,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:51,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:51,285 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:51,285 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:51,285 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:51,285 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:51,285 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:51,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:51,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:51,285 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:51,345 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=655 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/624d59e44fe64ab6a8cfbfa1799b629d 2023-04-09 09:55:51,348 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/6567c7954953486e8877579f00b1bc34 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/6567c7954953486e8877579f00b1bc34 2023-04-09 09:55:51,352 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/6567c7954953486e8877579f00b1bc34, entries=250, sequenceid=655, filesize=16.7 K 2023-04-09 09:55:51,352 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/b1d903ec662641b29368389476396ea4 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/b1d903ec662641b29368389476396ea4 2023-04-09 09:55:51,355 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/b1d903ec662641b29368389476396ea4, entries=150, sequenceid=655, filesize=11.9 K 2023-04-09 09:55:51,356 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/624d59e44fe64ab6a8cfbfa1799b629d as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/624d59e44fe64ab6a8cfbfa1799b629d 2023-04-09 09:55:51,359 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/624d59e44fe64ab6a8cfbfa1799b629d, entries=150, sequenceid=655, filesize=11.9 K 2023-04-09 09:55:51,360 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 2764d7f8155e1e5b0d641eebe063fa00 in 1250ms, sequenceid=655, compaction requested=true 2023-04-09 09:55:51,360 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2764d7f8155e1e5b0d641eebe063fa00: 2023-04-09 09:55:51,360 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-09 09:55:51,360 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:55:51,360 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:55:51,360 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:55:51,360 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-09 09:55:51,361 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-09 09:55:51,361 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-09 09:55:51,361 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. because compaction request was cancelled 2023-04-09 09:55:51,361 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-09 09:55:51,361 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 61796 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-09 09:55:51,362 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1912): 2764d7f8155e1e5b0d641eebe063fa00/A is initiating minor compaction (all files) 2023-04-09 09:55:51,362 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2764d7f8155e1e5b0d641eebe063fa00/A in TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:51,362 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/b1e58e24eef54e16bc7b0a877bf2d790, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/230a11401f0441d29e6c8454dbba3b8c, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/7c9e3ced0699439fb67d36440eaef12c, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/6567c7954953486e8877579f00b1bc34] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp, totalSize=60.3 K 2023-04-09 09:55:51,362 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting b1e58e24eef54e16bc7b0a877bf2d790, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=584, earliestPutTs=1681034149518 2023-04-09 09:55:51,362 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-09 09:55:51,362 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-09 09:55:51,362 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. because compaction request was cancelled 2023-04-09 09:55:51,362 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 230a11401f0441d29e6c8454dbba3b8c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=619, earliestPutTs=1681034149919 2023-04-09 09:55:51,363 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 7c9e3ced0699439fb67d36440eaef12c, keycount=300, bloomtype=ROW, size=19.0 K, encoding=NONE, compression=NONE, seqNum=635, earliestPutTs=1681034149930 2023-04-09 09:55:51,363 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 6567c7954953486e8877579f00b1bc34, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=655, earliestPutTs=1681034150091 2023-04-09 09:55:51,370 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2764d7f8155e1e5b0d641eebe063fa00#A#compaction#506 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:55:51,384 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/ade3157ea4824c0e9be3f1e14883f4f5 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/ade3157ea4824c0e9be3f1e14883f4f5 2023-04-09 09:55:51,385 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:51,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:51,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:51,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:51,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:51,389 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:51,389 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2764d7f8155e1e5b0d641eebe063fa00/A of 2764d7f8155e1e5b0d641eebe063fa00 into ade3157ea4824c0e9be3f1e14883f4f5(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:55:51,389 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:51,389 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2764d7f8155e1e5b0d641eebe063fa00: 2023-04-09 09:55:51,389 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:51,389 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:51,389 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00., storeName=2764d7f8155e1e5b0d641eebe063fa00/A, priority=12, startTime=1681034151360; duration=0sec 2023-04-09 09:55:51,390 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:55:51,390 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:51,390 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:51,391 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,391 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:51,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:51,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:51,391 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,391 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:51,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:51,392 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:51,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:51,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:51,393 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:51,393 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:51,393 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:51,394 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:51,394 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:51,394 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:51,394 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:51,394 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:51,394 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:51,394 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:51,394 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2764d7f8155e1e5b0d641eebe063fa00 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-09 09:55:51,394 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=A 2023-04-09 09:55:51,394 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:51,394 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=B 2023-04-09 09:55:51,394 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:51,395 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=C 2023-04-09 09:55:51,395 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:51,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 2764d7f8155e1e5b0d641eebe063fa00 2023-04-09 09:55:51,632 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. as already flushing 2023-04-09 09:55:51,681 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:51,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59328 deadline: 1681034211677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,681 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:51,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59302 deadline: 1681034211681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,784 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:51,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59328 deadline: 1681034211782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,785 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:51,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59302 deadline: 1681034211782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,801 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=671 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/20e537b474aa44c588ad3d69732713f2 2023-04-09 09:55:51,810 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=671 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/2b319e69193542d19e321b63f217cf6d 2023-04-09 09:55:51,818 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=671 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/dcb64d88866344aaba909576b211ac7f 2023-04-09 09:55:51,821 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/20e537b474aa44c588ad3d69732713f2 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/20e537b474aa44c588ad3d69732713f2 2023-04-09 09:55:51,824 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/20e537b474aa44c588ad3d69732713f2, entries=150, sequenceid=671, filesize=11.9 K 2023-04-09 09:55:51,825 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/2b319e69193542d19e321b63f217cf6d as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/2b319e69193542d19e321b63f217cf6d 2023-04-09 09:55:51,828 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/2b319e69193542d19e321b63f217cf6d, entries=150, sequenceid=671, filesize=11.9 K 2023-04-09 09:55:51,829 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/dcb64d88866344aaba909576b211ac7f as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/dcb64d88866344aaba909576b211ac7f 2023-04-09 09:55:51,831 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/dcb64d88866344aaba909576b211ac7f, entries=150, sequenceid=671, filesize=11.9 K 2023-04-09 09:55:51,832 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 2764d7f8155e1e5b0d641eebe063fa00 in 438ms, sequenceid=671, compaction requested=true 2023-04-09 09:55:51,832 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2764d7f8155e1e5b0d641eebe063fa00: 2023-04-09 09:55:51,832 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:51,832 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:51,832 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:51,832 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,832 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:51,832 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:51,834 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,834 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:51,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:51,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:51,834 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:51,834 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:51,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:51,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:51,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:51,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:51,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:51,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@526d773b[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:51,835 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:51,835 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:51,836 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:51,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:51,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:51,836 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:51,837 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:51,837 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:51,837 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,837 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:51,844 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:51,844 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,844 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:51,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:51,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:51,844 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:51,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:51,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:51,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:51,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:51,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:51,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:51,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:51,846 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:51,846 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:51,846 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:51,846 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:51,846 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:51,847 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:51,847 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:51,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:51,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:51,847 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:51,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:51,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:51,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:51,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:51,848 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:51,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:51,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:51,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:51,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:51,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:51,849 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:51,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:51,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,851 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:51,851 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:51,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:51,852 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:51,851 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:51,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:51,851 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:51,852 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:51,852 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:51,851 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:51,852 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:51,852 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,852 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:51,852 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:51,852 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:51,852 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:51,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:51,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:51,952 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:51,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:51,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:51,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:51,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:51,955 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:51,955 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:51,955 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:51,955 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:51,956 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:51,956 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:51,957 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:51,957 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:51,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:51,957 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:51,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:51,957 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:51,958 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:51,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:51,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:51,958 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:51,959 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:51,959 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:51,959 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:51,959 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:51,959 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:51,959 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:51,959 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:51,959 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:51,959 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:51,959 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2764d7f8155e1e5b0d641eebe063fa00 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-09 09:55:51,959 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=A 2023-04-09 09:55:51,960 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:51,960 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=B 2023-04-09 09:55:51,960 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:51,960 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=C 2023-04-09 09:55:51,960 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:51,966 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=691 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/c784667685e04953836405e3c674c294 2023-04-09 09:55:51,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 2764d7f8155e1e5b0d641eebe063fa00 2023-04-09 09:55:51,991 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. as already flushing 2023-04-09 09:55:51,993 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=691 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/1c5e7f749b314e55a701927c4096b029 2023-04-09 09:55:52,021 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:52,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59328 deadline: 1681034212014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:52,021 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:52,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59302 deadline: 1681034212018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:52,126 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:52,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59328 deadline: 1681034212122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:52,127 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:52,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59302 deadline: 1681034212122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:52,231 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:52,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59344 deadline: 1681034212228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:52,288 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:52,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59316 deadline: 1681034212286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:52,288 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:52,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59348 deadline: 1681034212287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:52,332 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:52,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59328 deadline: 1681034212328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:52,332 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:52,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59302 deadline: 1681034212328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:52,420 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=691 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/a0e4a221fc854a0d946a1923019b46df 2023-04-09 09:55:52,423 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/c784667685e04953836405e3c674c294 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/c784667685e04953836405e3c674c294 2023-04-09 09:55:52,426 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/c784667685e04953836405e3c674c294, entries=150, sequenceid=691, filesize=11.9 K 2023-04-09 09:55:52,427 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/1c5e7f749b314e55a701927c4096b029 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/1c5e7f749b314e55a701927c4096b029 2023-04-09 09:55:52,430 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/1c5e7f749b314e55a701927c4096b029, entries=150, sequenceid=691, filesize=11.9 K 2023-04-09 09:55:52,430 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/a0e4a221fc854a0d946a1923019b46df as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/a0e4a221fc854a0d946a1923019b46df 2023-04-09 09:55:52,434 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/a0e4a221fc854a0d946a1923019b46df, entries=150, sequenceid=691, filesize=11.9 K 2023-04-09 09:55:52,434 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 2764d7f8155e1e5b0d641eebe063fa00 in 475ms, sequenceid=691, compaction requested=true 2023-04-09 09:55:52,434 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2764d7f8155e1e5b0d641eebe063fa00: 2023-04-09 09:55:52,434 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:52,434 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:52,435 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:52,435 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:52,435 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:52,435 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:52,436 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:52,436 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:52,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:52,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:52,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:52,436 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:52,436 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:52,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:52,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:52,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:52,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:52,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:52,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:52,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74223cdc[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:52,438 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:52,438 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:52,439 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:52,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:52,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:52,439 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:52,439 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:52,439 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:52,439 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:52,439 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:52,440 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:52,440 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:52,440 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:52,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:52,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:52,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:52,440 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:52,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:52,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:52,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:52,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:52,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:52,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:52,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:52,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:52,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:52,442 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:52,443 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:52,443 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:52,443 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:52,443 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:52,452 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:52,452 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:52,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:52,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:52,452 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:52,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:52,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:52,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:52,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:52,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:52,452 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:52,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:52,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:52,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:52,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:52,453 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:52,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:52,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:52,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:52,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:52,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:52,455 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:52,455 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:52,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:52,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:52,455 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:52,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:52,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:52,456 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:52,455 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:52,456 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:52,456 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:52,456 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:52,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:52,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:52,456 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:52,456 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:52,456 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:52,456 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:52,456 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:52,556 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:52,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:52,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:52,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:52,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:52,559 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:52,559 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:52,559 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:52,559 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:52,560 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:52,560 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:52,561 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:52,561 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:52,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:52,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:52,561 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:52,561 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:52,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:52,562 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:52,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:52,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:52,562 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:52,562 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:52,562 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:52,562 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:52,562 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:52,562 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:52,562 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:52,562 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:52,563 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:52,563 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:52,563 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2764d7f8155e1e5b0d641eebe063fa00 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-09 09:55:52,563 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=A 2023-04-09 09:55:52,563 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:52,563 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=B 2023-04-09 09:55:52,563 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:52,563 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=C 2023-04-09 09:55:52,563 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:52,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 2764d7f8155e1e5b0d641eebe063fa00 2023-04-09 09:55:52,637 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. as already flushing 2023-04-09 09:55:52,696 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:52,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59328 deadline: 1681034212690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:52,704 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:52,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59302 deadline: 1681034212697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:52,802 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:52,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59328 deadline: 1681034212798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:52,809 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:52,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59302 deadline: 1681034212805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:52,985 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=703 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/de4eec7029aa4036bd620a99d4fec7ce 2023-04-09 09:55:53,013 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:53,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59328 deadline: 1681034213005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,014 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:53,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59302 deadline: 1681034213011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,022 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=703 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/77bdac29ac504c649732a970de58ce3d 2023-04-09 09:55:53,042 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=703 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/bb4e2f5bbc3849b9a3e94c5ca7cff084 2023-04-09 09:55:53,045 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/de4eec7029aa4036bd620a99d4fec7ce as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/de4eec7029aa4036bd620a99d4fec7ce 2023-04-09 09:55:53,049 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/de4eec7029aa4036bd620a99d4fec7ce, entries=150, sequenceid=703, filesize=11.9 K 2023-04-09 09:55:53,049 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/77bdac29ac504c649732a970de58ce3d as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/77bdac29ac504c649732a970de58ce3d 2023-04-09 09:55:53,052 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/77bdac29ac504c649732a970de58ce3d, entries=150, sequenceid=703, filesize=11.9 K 2023-04-09 09:55:53,053 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/bb4e2f5bbc3849b9a3e94c5ca7cff084 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/bb4e2f5bbc3849b9a3e94c5ca7cff084 2023-04-09 09:55:53,056 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/bb4e2f5bbc3849b9a3e94c5ca7cff084, entries=150, sequenceid=703, filesize=11.9 K 2023-04-09 09:55:53,056 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 2764d7f8155e1e5b0d641eebe063fa00 in 493ms, sequenceid=703, compaction requested=true 2023-04-09 09:55:53,056 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2764d7f8155e1e5b0d641eebe063fa00: 2023-04-09 09:55:53,056 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:53,056 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:53,056 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:53,057 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,057 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:53,057 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:53,058 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,058 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:53,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:53,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:53,059 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:53,059 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:53,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:53,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:53,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:53,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:53,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:53,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c54c902[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:53,060 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:53,060 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:53,065 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:53,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:53,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:53,065 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:53,065 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:53,065 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:53,065 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,065 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:53,066 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:53,066 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:53,066 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,066 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:53,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:53,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:53,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:53,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:53,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:53,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:53,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:53,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:53,068 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:53,068 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:53,068 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:53,068 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:53,068 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:53,068 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:53,071 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:53,071 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:53,071 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:53,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:53,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:53,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:53,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:53,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:53,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:53,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:53,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:53,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:53,072 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:53,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:53,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:53,073 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:53,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:53,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,075 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:53,075 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:53,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:53,075 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,075 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:53,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:53,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:53,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:53,075 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:53,075 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:53,075 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,075 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:53,075 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:53,075 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:53,075 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:53,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:53,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:53,075 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:53,176 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:53,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:53,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:53,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:53,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:53,179 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:53,179 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:53,179 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:53,179 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:53,179 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:53,179 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:53,183 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,183 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:53,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:53,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:53,183 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,184 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:53,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:53,184 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:53,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:53,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:53,184 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:53,184 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:53,184 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:53,184 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:53,185 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:53,185 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:53,185 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:53,185 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:53,185 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:53,185 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:53,185 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2764d7f8155e1e5b0d641eebe063fa00 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-09 09:55:53,185 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=A 2023-04-09 09:55:53,185 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:53,185 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=B 2023-04-09 09:55:53,185 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:53,185 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=C 2023-04-09 09:55:53,185 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:53,191 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=723 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/32fa6ba826f146f29b7968d35092b4e3 2023-04-09 09:55:53,200 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=723 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/7dd5a2bf098f4ee9b9583cec2649dc38 2023-04-09 09:55:53,209 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=723 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/e25e526b7a0e4350a756e977f35c6ceb 2023-04-09 09:55:53,212 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/32fa6ba826f146f29b7968d35092b4e3 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/32fa6ba826f146f29b7968d35092b4e3 2023-04-09 09:55:53,215 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/32fa6ba826f146f29b7968d35092b4e3, entries=150, sequenceid=723, filesize=11.9 K 2023-04-09 09:55:53,216 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/7dd5a2bf098f4ee9b9583cec2649dc38 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/7dd5a2bf098f4ee9b9583cec2649dc38 2023-04-09 09:55:53,220 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/7dd5a2bf098f4ee9b9583cec2649dc38, entries=150, sequenceid=723, filesize=11.9 K 2023-04-09 09:55:53,220 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/e25e526b7a0e4350a756e977f35c6ceb as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/e25e526b7a0e4350a756e977f35c6ceb 2023-04-09 09:55:53,223 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/e25e526b7a0e4350a756e977f35c6ceb, entries=150, sequenceid=723, filesize=11.9 K 2023-04-09 09:55:53,224 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for 2764d7f8155e1e5b0d641eebe063fa00 in 39ms, sequenceid=723, compaction requested=true 2023-04-09 09:55:53,224 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2764d7f8155e1e5b0d641eebe063fa00: 2023-04-09 09:55:53,224 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:53,224 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:53,224 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:53,224 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,224 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:53,224 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:53,225 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,225 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:53,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:53,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:53,226 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:53,226 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:53,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:53,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:53,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:53,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:53,227 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:53,227 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e96358f[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:53,227 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:53,227 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:53,228 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:53,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:53,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:53,228 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,228 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:53,228 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:53,228 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:53,229 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:53,231 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,231 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:53,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:53,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:53,231 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:53,231 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:53,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:53,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:53,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:53,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:53,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:53,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:53,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:53,233 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:53,233 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:53,233 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:53,233 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:53,233 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:53,236 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:53,236 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:53,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:53,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:53,236 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:53,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:53,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:53,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:53,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:53,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:53,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:53,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:53,237 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:53,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:53,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:53,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,238 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:53,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:53,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,240 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:53,240 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:53,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:53,241 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:53,240 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:53,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:53,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:53,241 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:53,240 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,241 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:53,241 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:53,241 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:53,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:53,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:53,241 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,241 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:53,241 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:53,241 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:53,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 2764d7f8155e1e5b0d641eebe063fa00 2023-04-09 09:55:53,330 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2764d7f8155e1e5b0d641eebe063fa00 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-09 09:55:53,330 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=A 2023-04-09 09:55:53,330 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:53,330 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=B 2023-04-09 09:55:53,330 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:53,330 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=C 2023-04-09 09:55:53,330 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:53,341 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:53,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:53,343 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:53,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:53,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:53,344 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:53,344 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:53,344 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:53,344 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:53,344 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:53,345 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:53,346 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,346 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:53,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:53,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:53,346 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,346 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:53,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:53,346 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:53,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:53,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:53,347 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:53,347 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:53,347 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:53,347 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:53,347 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:53,347 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:53,347 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:53,347 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:53,347 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:53,347 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:53,347 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. as already flushing 2023-04-09 09:55:53,347 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:53,348 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:53,348 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:53,348 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,348 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:53,348 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:53,349 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:53,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:53,349 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:53,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:53,350 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:53,350 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:53,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:53,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:53,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:53,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:53,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@377b5fc9[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:53,351 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:53,351 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:53,358 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:53,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:53,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:53,358 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:53,358 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:53,358 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:53,358 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,358 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:53,359 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:53,359 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:53,359 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:53,359 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:53,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:53,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:53,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:53,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:53,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:53,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:53,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:53,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:53,367 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:53,367 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:53,367 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:53,367 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:53,367 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:53,373 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:53,373 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:53,373 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:53,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:53,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:53,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:53,373 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:53,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:53,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:53,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:53,375 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:53,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:53,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:53,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:53,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:53,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:53,380 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,380 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:53,380 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:53,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:53,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:53,381 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:53,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:53,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:53,381 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:53,381 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:53,381 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:53,381 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,381 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:53,381 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:53,381 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:53,381 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:53,409 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034153377, max timestamp=1681034153408Num uniques 859; 2023-04-09 09:55:53,409 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:55:53,410 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034153377, max timestamp=1681034153408Num uniques 859; 2023-04-09 09:55:53,410 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034153377, max timestamp=1681034153408Num uniques 859; 2023-04-09 09:55:53,410 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:55:53,410 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:55:53,481 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:53,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:53,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:53,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:53,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:53,493 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:53,493 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:53,493 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:53,493 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:53,493 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:53,493 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:53,494 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:53,494 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:53,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:53,495 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:53,495 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:53,495 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:53,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:53,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:53,495 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:53,496 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:53,496 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:53,496 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:53,496 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:53,496 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:53,500 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:53,500 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:53,500 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:53,500 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:53,500 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. as already flushing 2023-04-09 09:55:53,500 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:53,500 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:53,500 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:53,500 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,500 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:53,500 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:53,501 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,501 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:53,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:53,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:53,502 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:53,502 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:53,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:53,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:53,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:53,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:53,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:53,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52da5126[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:53,503 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:53,503 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:53,504 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:53,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:53,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:53,504 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:53,504 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:53,504 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:53,510 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:53,510 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:53,510 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:53,510 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,511 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:53,511 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:53,511 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:53,511 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:53,511 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:53,511 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:53,511 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:53,512 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:53,512 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:53,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:53,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:53,512 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:53,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:53,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:53,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:53,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:53,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:53,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:53,513 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:53,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:53,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:53,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:53,514 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:53,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:53,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,516 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,516 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:53,516 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:53,516 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:53,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:53,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:53,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:53,516 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:53,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:53,516 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:53,516 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:53,516 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:53,516 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,516 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:53,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:53,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:53,516 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:53,516 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:53,516 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:53,520 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:53,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59302 deadline: 1681034213505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,529 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:53,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59328 deadline: 1681034213518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,616 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:53,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:53,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:53,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:53,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:53,619 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:53,619 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:53,620 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:53,620 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:53,620 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:53,620 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:53,621 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:53,621 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:53,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:53,622 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,622 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:53,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:53,622 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:53,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:53,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:53,622 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:53,622 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:53,623 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:53,623 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:53,623 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:53,623 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:53,623 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:53,623 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:53,623 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:53,623 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:53,623 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. as already flushing 2023-04-09 09:55:53,623 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:53,623 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:53,623 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:53,623 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,623 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:53,623 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:53,625 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:53,625 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:53,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:53,626 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:53,626 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:53,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:53,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:53,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:53,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:53,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:53,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78ec8ea6[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:53,627 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:53,627 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:53,628 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:53,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:53,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:53,628 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:53,628 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:53,628 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:53,628 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,628 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:53,629 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:53,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:53,629 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:53,630 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:53,630 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:53,630 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:53,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59302 deadline: 1681034213621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:53,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:53,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:53,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:53,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:53,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:53,632 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:53,632 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:53,632 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:53,632 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:53,632 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:53,632 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:53,633 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:53,633 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:53,633 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:53,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:53,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:53,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:53,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:53,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:53,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:53,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:53,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:53,634 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:53,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:53,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:53,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:53,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,635 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:53,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:53,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,638 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:53,638 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:53,638 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:53,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:53,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:53,638 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:53,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:53,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:53,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59328 deadline: 1681034213630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,638 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:53,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:53,639 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:53,639 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:53,639 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:53,639 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:53,639 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,639 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:53,639 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:53,639 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:53,736 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=733 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/9046d34cd4b84d99b54cfa0b62d7b1c1 2023-04-09 09:55:53,739 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:53,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:53,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:53,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:53,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:53,742 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:53,742 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:53,742 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:53,742 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:53,743 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:53,743 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:53,744 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:53,744 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:53,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:53,744 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,744 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:53,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:53,745 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:53,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:53,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:53,745 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:53,745 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:53,745 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:53,745 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:53,745 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:53,746 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:53,746 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:53,746 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:53,746 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:53,746 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:53,746 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. as already flushing 2023-04-09 09:55:53,746 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:53,746 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:53,746 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:53,746 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,746 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:53,746 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:53,747 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:53,748 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:53,748 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:53,748 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:53,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:53,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:53,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:53,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:53,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:53,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:53,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47e9e679[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:53,750 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:53,751 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:53,751 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:53,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:53,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:53,752 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,752 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:53,752 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:53,752 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:53,752 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:53,754 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:53,755 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:53,755 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,755 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:53,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:53,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:53,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:53,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:53,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:53,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:53,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:53,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:53,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:53,757 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:53,757 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:53,757 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:53,757 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:53,757 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:53,758 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:53,758 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:53,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:53,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:53,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:53,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:53,758 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:53,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:53,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:53,759 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:53,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:53,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:53,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:53,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:53,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:53,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,760 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:53,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:53,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,764 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,764 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:53,764 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:53,764 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:53,765 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:53,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:53,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:53,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:53,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:53,765 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,765 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:53,765 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:53,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:53,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:53,765 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:53,765 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:53,765 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:53,766 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:53,839 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:53,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59302 deadline: 1681034213832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,843 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:53,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59328 deadline: 1681034213840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,866 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:53,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:53,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:53,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:53,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:53,869 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:53,869 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:53,869 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:53,869 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:53,869 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:53,869 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:53,871 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:53,871 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:53,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:53,872 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,872 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:53,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:53,872 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:53,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:53,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:53,872 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:53,873 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:53,873 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:53,873 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:53,873 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:53,873 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:53,873 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:53,873 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:53,873 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:53,873 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:53,873 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. as already flushing 2023-04-09 09:55:53,873 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:53,873 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:53,873 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:53,873 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,873 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:53,873 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:53,875 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,875 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:53,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:53,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:53,875 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:53,875 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:53,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:53,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:53,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:53,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:53,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:53,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a016c5a[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:53,876 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:53,876 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:53,877 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:53,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:53,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:53,878 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:53,878 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:53,878 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,878 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:53,878 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:53,879 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,879 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:53,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:53,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:53,879 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:53,879 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:53,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:53,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:53,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:53,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:53,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:53,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:53,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:53,881 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:53,881 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:53,881 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:53,881 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:53,881 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:53,883 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:53,883 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:53,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:53,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:53,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:53,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:53,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:53,883 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:53,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:53,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:53,884 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:53,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:53,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:53,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:53,885 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:53,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:53,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:53,887 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,887 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:53,887 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:53,887 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:53,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:53,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:53,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:53,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:53,887 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:53,887 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:53,887 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:53,887 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:53,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:53,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:53,887 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,887 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:53,887 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:53,887 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:53,887 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:53,987 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:53,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:53,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:53,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:53,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:53,990 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:53,990 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:53,990 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:53,990 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:53,991 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:53,991 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:53,992 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:53,992 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:53,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:53,992 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,992 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:53,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:53,993 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:53,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:53,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:53,993 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:53,993 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:53,993 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:53,993 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:53,993 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:53,993 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:53,994 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:53,994 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:53,994 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:53,994 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:53,994 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. as already flushing 2023-04-09 09:55:53,994 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:53,994 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:53,994 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:53,994 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,994 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:53,994 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:53,995 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,995 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:53,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:53,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:53,996 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:53,996 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:53,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:53,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:53,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:53,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:53,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:53,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:53,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a524b5d[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:53,997 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:53,997 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:53,999 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:53,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:53,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:53,999 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:53,999 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:53,999 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:53,999 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:54,000 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:54,001 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:54,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:54,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:54,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:54,001 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:54,001 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:54,001 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:54,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:54,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:54,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:54,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:54,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:54,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:54,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:54,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:54,003 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:54,003 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:54,003 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:54,003 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:54,003 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:54,003 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:54,004 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:54,004 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:54,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:54,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:54,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:54,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:54,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:54,004 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:54,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:54,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:54,005 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:54,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:54,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:54,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:54,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:54,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:54,006 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:54,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:54,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:54,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:54,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:54,008 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:54,008 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:54,008 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:54,008 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:54,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:54,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:54,008 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:54,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:54,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:54,008 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:54,008 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:54,009 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:54,008 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:54,008 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:54,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:54,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:54,009 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:54,009 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:54,009 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:54,109 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:54,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:54,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:54,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:54,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:54,112 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:54,112 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:54,112 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:54,112 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:54,113 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:54,113 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:54,115 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:54,115 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:54,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:54,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:54,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:54,115 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:54,116 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:54,116 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:54,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:54,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:54,116 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:54,116 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:54,116 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:54,116 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:54,116 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:54,117 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:54,117 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:54,117 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:54,117 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:54,117 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:54,117 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. as already flushing 2023-04-09 09:55:54,117 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:54,117 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:54,117 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:54,117 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:54,117 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:54,117 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:54,118 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:54,118 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:54,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:54,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:54,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:54,119 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:54,119 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:54,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:54,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:54,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:54,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:54,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:54,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:54,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3657790f[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:54,120 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:54,120 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:54,121 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:54,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:54,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:54,121 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:54,121 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:54,121 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:54,121 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:54,121 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:54,122 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:54,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:54,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:54,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:54,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:54,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:54,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:54,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:54,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:54,124 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:54,124 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:54,124 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:54,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:54,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:54,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:54,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:54,125 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:54,125 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:54,125 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:54,125 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:54,125 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:54,127 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:54,127 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:54,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:54,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:54,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:54,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:54,128 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:54,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:54,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:54,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:54,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:54,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:54,129 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:54,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:54,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:54,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:54,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:54,130 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:54,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:54,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:54,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:54,132 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:54,132 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:54,132 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:54,132 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:54,132 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:54,132 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:54,132 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:54,132 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:54,132 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:54,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:54,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:54,132 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:54,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:54,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:54,133 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:54,133 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:54,133 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:54,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:54,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:54,147 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=733 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/f938136ac3e747d0976e0aab2c3e7236 2023-04-09 09:55:54,148 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:54,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59302 deadline: 1681034214141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:54,152 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:54,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59328 deadline: 1681034214146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:54,155 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=733 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/a8b018d403ce4ddb9d9c3bbce17b6552 2023-04-09 09:55:54,158 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/9046d34cd4b84d99b54cfa0b62d7b1c1 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/9046d34cd4b84d99b54cfa0b62d7b1c1 2023-04-09 09:55:54,161 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/9046d34cd4b84d99b54cfa0b62d7b1c1, entries=250, sequenceid=733, filesize=16.7 K 2023-04-09 09:55:54,162 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/f938136ac3e747d0976e0aab2c3e7236 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/f938136ac3e747d0976e0aab2c3e7236 2023-04-09 09:55:54,164 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/f938136ac3e747d0976e0aab2c3e7236, entries=150, sequenceid=733, filesize=11.9 K 2023-04-09 09:55:54,165 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/a8b018d403ce4ddb9d9c3bbce17b6552 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/a8b018d403ce4ddb9d9c3bbce17b6552 2023-04-09 09:55:54,168 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/a8b018d403ce4ddb9d9c3bbce17b6552, entries=150, sequenceid=733, filesize=11.9 K 2023-04-09 09:55:54,168 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 2764d7f8155e1e5b0d641eebe063fa00 in 838ms, sequenceid=733, compaction requested=true 2023-04-09 09:55:54,168 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2764d7f8155e1e5b0d641eebe063fa00: 2023-04-09 09:55:54,169 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:55:54,169 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-09 09:55:54,169 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-09 09:55:54,169 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:55:54,169 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:55:54,170 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 78984 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-09 09:55:54,170 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1912): 2764d7f8155e1e5b0d641eebe063fa00/A is initiating minor compaction (all files) 2023-04-09 09:55:54,170 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2764d7f8155e1e5b0d641eebe063fa00/A in TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:54,170 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/ade3157ea4824c0e9be3f1e14883f4f5, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/20e537b474aa44c588ad3d69732713f2, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/c784667685e04953836405e3c674c294, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/de4eec7029aa4036bd620a99d4fec7ce, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/32fa6ba826f146f29b7968d35092b4e3, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/9046d34cd4b84d99b54cfa0b62d7b1c1] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp, totalSize=77.1 K 2023-04-09 09:55:54,171 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 86256 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-09 09:55:54,171 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): 2764d7f8155e1e5b0d641eebe063fa00/B is initiating minor compaction (all files) 2023-04-09 09:55:54,171 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2764d7f8155e1e5b0d641eebe063fa00/B in TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:54,171 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting ade3157ea4824c0e9be3f1e14883f4f5, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=655, earliestPutTs=1681034150092 2023-04-09 09:55:54,171 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/df01f61a1ec1429ea0b551f4665aca74, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/b1d903ec662641b29368389476396ea4, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/2b319e69193542d19e321b63f217cf6d, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/1c5e7f749b314e55a701927c4096b029, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/77bdac29ac504c649732a970de58ce3d, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/7dd5a2bf098f4ee9b9583cec2649dc38, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/f938136ac3e747d0976e0aab2c3e7236] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp, totalSize=84.2 K 2023-04-09 09:55:54,171 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 20e537b474aa44c588ad3d69732713f2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=671, earliestPutTs=1681034150131 2023-04-09 09:55:54,171 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting df01f61a1ec1429ea0b551f4665aca74, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=635, earliestPutTs=1681034150052 2023-04-09 09:55:54,172 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting c784667685e04953836405e3c674c294, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=691, earliestPutTs=1681034151666 2023-04-09 09:55:54,172 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting b1d903ec662641b29368389476396ea4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=655, earliestPutTs=1681034150092 2023-04-09 09:55:54,172 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting de4eec7029aa4036bd620a99d4fec7ce, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=703, earliestPutTs=1681034152006 2023-04-09 09:55:54,172 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 2b319e69193542d19e321b63f217cf6d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=671, earliestPutTs=1681034150131 2023-04-09 09:55:54,172 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 32fa6ba826f146f29b7968d35092b4e3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=723, earliestPutTs=1681034152675 2023-04-09 09:55:54,172 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 1c5e7f749b314e55a701927c4096b029, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=691, earliestPutTs=1681034151666 2023-04-09 09:55:54,172 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 9046d34cd4b84d99b54cfa0b62d7b1c1, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=733, earliestPutTs=1681034153318 2023-04-09 09:55:54,173 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 77bdac29ac504c649732a970de58ce3d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=703, earliestPutTs=1681034152006 2023-04-09 09:55:54,174 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 7dd5a2bf098f4ee9b9583cec2649dc38, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=723, earliestPutTs=1681034152675 2023-04-09 09:55:54,175 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting f938136ac3e747d0976e0aab2c3e7236, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=733, earliestPutTs=1681034153324 2023-04-09 09:55:54,183 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2764d7f8155e1e5b0d641eebe063fa00#A#compaction#522 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:55:54,190 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): 2764d7f8155e1e5b0d641eebe063fa00#B#compaction#523 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:55:54,197 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/accb65ed4e9348c2ac901af52ed54730 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/accb65ed4e9348c2ac901af52ed54730 2023-04-09 09:55:54,202 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 2764d7f8155e1e5b0d641eebe063fa00/A of 2764d7f8155e1e5b0d641eebe063fa00 into accb65ed4e9348c2ac901af52ed54730(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:55:54,202 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2764d7f8155e1e5b0d641eebe063fa00: 2023-04-09 09:55:54,202 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00., storeName=2764d7f8155e1e5b0d641eebe063fa00/A, priority=10, startTime=1681034154168; duration=0sec 2023-04-09 09:55:54,202 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:55:54,202 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-09 09:55:54,204 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 86256 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-09 09:55:54,204 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1912): 2764d7f8155e1e5b0d641eebe063fa00/C is initiating minor compaction (all files) 2023-04-09 09:55:54,204 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2764d7f8155e1e5b0d641eebe063fa00/C in TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:54,204 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/f45909998c0d4fc7b46e9ee0e347901d, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/624d59e44fe64ab6a8cfbfa1799b629d, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/dcb64d88866344aaba909576b211ac7f, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/a0e4a221fc854a0d946a1923019b46df, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/bb4e2f5bbc3849b9a3e94c5ca7cff084, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/e25e526b7a0e4350a756e977f35c6ceb, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/a8b018d403ce4ddb9d9c3bbce17b6552] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp, totalSize=84.2 K 2023-04-09 09:55:54,204 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting f45909998c0d4fc7b46e9ee0e347901d, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=635, earliestPutTs=1681034150052 2023-04-09 09:55:54,205 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 624d59e44fe64ab6a8cfbfa1799b629d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=655, earliestPutTs=1681034150092 2023-04-09 09:55:54,205 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting dcb64d88866344aaba909576b211ac7f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=671, earliestPutTs=1681034150131 2023-04-09 09:55:54,205 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting a0e4a221fc854a0d946a1923019b46df, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=691, earliestPutTs=1681034151666 2023-04-09 09:55:54,205 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting bb4e2f5bbc3849b9a3e94c5ca7cff084, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=703, earliestPutTs=1681034152006 2023-04-09 09:55:54,205 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting e25e526b7a0e4350a756e977f35c6ceb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=723, earliestPutTs=1681034152675 2023-04-09 09:55:54,206 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting a8b018d403ce4ddb9d9c3bbce17b6552, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=733, earliestPutTs=1681034153324 2023-04-09 09:55:54,215 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2764d7f8155e1e5b0d641eebe063fa00#C#compaction#524 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:55:54,224 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/51a816e2764b4fba8ef0a6119b45e0a6 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/51a816e2764b4fba8ef0a6119b45e0a6 2023-04-09 09:55:54,228 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 2764d7f8155e1e5b0d641eebe063fa00/C of 2764d7f8155e1e5b0d641eebe063fa00 into 51a816e2764b4fba8ef0a6119b45e0a6(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:55:54,228 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2764d7f8155e1e5b0d641eebe063fa00: 2023-04-09 09:55:54,228 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00., storeName=2764d7f8155e1e5b0d641eebe063fa00/C, priority=9, startTime=1681034154169; duration=0sec 2023-04-09 09:55:54,228 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:55:54,233 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:54,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:54,235 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:54,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:54,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:54,236 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:54,236 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:54,236 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:54,236 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:54,237 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:54,237 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:54,238 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:54,238 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:54,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:54,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:54,238 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:54,238 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:54,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:54,239 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:54,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:54,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:54,239 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:54,239 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:54,240 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:54,240 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:54,240 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:54,240 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:54,240 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:54,240 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:54,240 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:54,240 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:54,240 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2764d7f8155e1e5b0d641eebe063fa00 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-09 09:55:54,240 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=A 2023-04-09 09:55:54,240 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:55:54,241 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=B 2023-04-09 09:55:54,241 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:55:54,241 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=C 2023-04-09 09:55:54,241 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:55:54,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 2764d7f8155e1e5b0d641eebe063fa00 2023-04-09 09:55:54,243 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. as already flushing 2023-04-09 09:55:54,287 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:54,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59344 deadline: 1681034214275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:54,297 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:54,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59316 deadline: 1681034214289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:54,299 DEBUG [Thread-3176] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4153 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00., hostname=jenkins-hbase4.apache.org,42467,1681034019544, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-09 09:55:54,300 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:54,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59348 deadline: 1681034214295, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:54,301 DEBUG [Thread-3180] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4150 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00., hostname=jenkins-hbase4.apache.org,42467,1681034019544, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-09 09:55:54,392 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:54,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59344 deadline: 1681034214388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:54,599 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/9081f967b0f849fa82ab4baa2d3e857c as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/9081f967b0f849fa82ab4baa2d3e857c 2023-04-09 09:55:54,600 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:54,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59344 deadline: 1681034214593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:54,602 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 2764d7f8155e1e5b0d641eebe063fa00/B of 2764d7f8155e1e5b0d641eebe063fa00 into 9081f967b0f849fa82ab4baa2d3e857c(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:55:54,603 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2764d7f8155e1e5b0d641eebe063fa00: 2023-04-09 09:55:54,603 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00., storeName=2764d7f8155e1e5b0d641eebe063fa00/B, priority=9, startTime=1681034154169; duration=0sec 2023-04-09 09:55:54,603 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:55:54,648 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=772 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/41534cc5172b4069b2743301a4081526 2023-04-09 09:55:54,656 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=772 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/18dce3b200ac48c796c51cc1d7421fc4 2023-04-09 09:55:54,658 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:54,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59302 deadline: 1681034214650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:54,659 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:54,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59328 deadline: 1681034214656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:54,905 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:54,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59344 deadline: 1681034214903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:55,065 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=772 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/5419ea9f45764d09b29444986d77f596 2023-04-09 09:55:55,068 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/41534cc5172b4069b2743301a4081526 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/41534cc5172b4069b2743301a4081526 2023-04-09 09:55:55,071 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/41534cc5172b4069b2743301a4081526, entries=150, sequenceid=772, filesize=11.9 K 2023-04-09 09:55:55,072 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/18dce3b200ac48c796c51cc1d7421fc4 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/18dce3b200ac48c796c51cc1d7421fc4 2023-04-09 09:55:55,075 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/18dce3b200ac48c796c51cc1d7421fc4, entries=150, sequenceid=772, filesize=11.9 K 2023-04-09 09:55:55,076 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/5419ea9f45764d09b29444986d77f596 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/5419ea9f45764d09b29444986d77f596 2023-04-09 09:55:55,079 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/5419ea9f45764d09b29444986d77f596, entries=150, sequenceid=772, filesize=11.9 K 2023-04-09 09:55:55,080 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 2764d7f8155e1e5b0d641eebe063fa00 in 839ms, sequenceid=772, compaction requested=false 2023-04-09 09:55:55,080 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2764d7f8155e1e5b0d641eebe063fa00: 2023-04-09 09:55:55,080 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:55,080 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:55,080 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:55,080 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:55,080 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:55,080 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:55,087 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:55,087 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:55,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:55,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:55,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:55,087 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:55,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:55,087 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:55,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:55,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:55,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:55,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:55,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:55,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e058b92[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:55,089 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:55,089 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:55,092 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:55,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:55,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:55,092 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:55,092 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:55,092 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:55,092 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:55,092 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:55,093 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:55,093 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:55,093 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:55,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:55,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:55,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:55,094 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:55,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:55,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:55,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:55,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:55,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:55,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:55,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:55,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:55,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:55,096 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:55,096 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:55,096 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:55,096 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:55,096 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:55,098 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:55,098 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:55,098 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:55,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:55,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:55,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:55,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:55,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:55,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:55,098 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:55,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:55,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:55,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:55,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:55,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:55,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:55,099 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:55,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:55,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:55,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:55,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:55,102 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:55,102 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:55,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:55,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:55,102 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:55,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:55,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:55,102 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:55,102 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:55,102 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:55,102 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:55,102 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:55,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:55,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:55,103 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:55,103 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:55,103 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:55,103 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:55,103 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:55,203 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:55,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:55,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:55,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:55,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:55,206 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:55,206 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:55,206 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:55,206 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:55,206 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:55,206 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:55,207 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:55,207 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:55,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:55,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:55,208 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:55,208 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:55,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:55,208 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:55,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:55,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:55,208 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:55,209 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:55,209 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:55,209 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:55,209 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:55,209 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:55,209 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:55,209 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:55,209 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:55,209 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:55,209 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2764d7f8155e1e5b0d641eebe063fa00 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-09 09:55:55,209 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=A 2023-04-09 09:55:55,209 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:55,209 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=B 2023-04-09 09:55:55,209 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:55,210 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=C 2023-04-09 09:55:55,210 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:55,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 2764d7f8155e1e5b0d641eebe063fa00 2023-04-09 09:55:55,410 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. as already flushing 2023-04-09 09:55:55,493 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034155425, max timestamp=1681034155491Num uniques 859; 2023-04-09 09:55:55,493 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034155425, max timestamp=1681034155491Num uniques 859; 2023-04-09 09:55:55,493 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:55:55,493 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034155425, max timestamp=1681034155491Num uniques 859; 2023-04-09 09:55:55,493 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:55:55,493 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:55:55,617 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=783 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/0f8e534b382a44f8b3393433187502dc 2023-04-09 09:55:55,619 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:55,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59344 deadline: 1681034215613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:55,665 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:55,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59328 deadline: 1681034215662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:55,668 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:55,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59302 deadline: 1681034215665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:55,722 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:55,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59344 deadline: 1681034215720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:55,927 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:55,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59344 deadline: 1681034215924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,026 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=783 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/c4659bdfb8844ef2a3da8aa1d8f86283 2023-04-09 09:55:56,035 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=783 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/56942368a0e648be9beef5ef67df0c07 2023-04-09 09:55:56,038 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/0f8e534b382a44f8b3393433187502dc as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/0f8e534b382a44f8b3393433187502dc 2023-04-09 09:55:56,041 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/0f8e534b382a44f8b3393433187502dc, entries=150, sequenceid=783, filesize=11.9 K 2023-04-09 09:55:56,042 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/c4659bdfb8844ef2a3da8aa1d8f86283 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/c4659bdfb8844ef2a3da8aa1d8f86283 2023-04-09 09:55:56,045 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/c4659bdfb8844ef2a3da8aa1d8f86283, entries=150, sequenceid=783, filesize=11.9 K 2023-04-09 09:55:56,045 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/56942368a0e648be9beef5ef67df0c07 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/56942368a0e648be9beef5ef67df0c07 2023-04-09 09:55:56,048 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/56942368a0e648be9beef5ef67df0c07, entries=150, sequenceid=783, filesize=11.9 K 2023-04-09 09:55:56,048 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 2764d7f8155e1e5b0d641eebe063fa00 in 839ms, sequenceid=783, compaction requested=true 2023-04-09 09:55:56,049 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2764d7f8155e1e5b0d641eebe063fa00: 2023-04-09 09:55:56,049 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:56,049 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:56,049 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:56,049 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,049 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:56,049 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:56,050 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:56,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:56,050 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:56,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:56,051 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:56,051 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:56,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:56,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:56,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:56,052 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:56,052 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1fc743b4[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:56,052 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:56,052 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:56,053 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:56,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:56,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:56,053 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:56,053 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:56,053 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,053 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:56,053 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:56,057 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:56,057 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:56,057 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,057 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:56,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:56,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:56,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:56,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:56,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:56,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:56,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:56,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:56,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:56,059 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:56,059 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:56,059 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:56,059 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:56,059 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:56,061 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:56,061 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:56,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:56,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:56,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:56,061 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:56,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:56,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:56,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:56,061 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:56,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:56,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:56,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:56,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:56,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:56,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,062 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:56,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:56,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,064 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:56,064 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:56,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:56,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:56,064 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,064 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:56,064 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:56,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:56,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:56,064 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:56,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:56,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:56,064 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,065 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:56,065 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:56,065 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:56,065 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:56,065 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:56,165 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:56,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:56,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:56,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:56,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:56,168 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:56,168 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:56,168 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:56,168 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:56,168 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:56,168 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:56,169 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:56,169 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:56,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:56,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:56,170 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,170 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:56,170 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:56,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:56,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:56,171 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:56,171 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:56,171 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:56,171 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:56,171 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:56,171 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:56,171 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:56,171 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:56,172 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:56,172 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:56,172 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2764d7f8155e1e5b0d641eebe063fa00 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-09 09:55:56,172 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=A 2023-04-09 09:55:56,172 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:55:56,172 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=B 2023-04-09 09:55:56,172 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:55:56,172 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=C 2023-04-09 09:55:56,172 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:55:56,183 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=820 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/48abc2ed111c4b50b033176785aa21cd 2023-04-09 09:55:56,195 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=820 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/c53eb5057bad4b9c8cc79653fa633d01 2023-04-09 09:55:56,207 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=820 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/155bbfdf6f3643caaedd0d4579ed5663 2023-04-09 09:55:56,213 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/48abc2ed111c4b50b033176785aa21cd as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/48abc2ed111c4b50b033176785aa21cd 2023-04-09 09:55:56,216 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/48abc2ed111c4b50b033176785aa21cd, entries=150, sequenceid=820, filesize=11.9 K 2023-04-09 09:55:56,217 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/c53eb5057bad4b9c8cc79653fa633d01 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/c53eb5057bad4b9c8cc79653fa633d01 2023-04-09 09:55:56,221 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/c53eb5057bad4b9c8cc79653fa633d01, entries=150, sequenceid=820, filesize=11.9 K 2023-04-09 09:55:56,221 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/155bbfdf6f3643caaedd0d4579ed5663 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/155bbfdf6f3643caaedd0d4579ed5663 2023-04-09 09:55:56,224 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/155bbfdf6f3643caaedd0d4579ed5663, entries=150, sequenceid=820, filesize=11.9 K 2023-04-09 09:55:56,224 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=0 B/0 for 2764d7f8155e1e5b0d641eebe063fa00 in 52ms, sequenceid=820, compaction requested=true 2023-04-09 09:55:56,224 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2764d7f8155e1e5b0d641eebe063fa00: 2023-04-09 09:55:56,224 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:56,225 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:56,225 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:56,225 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,225 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:56,225 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:56,227 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,227 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:56,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:56,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:56,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:56,227 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:56,228 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:56,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:56,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:56,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:56,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:56,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27f7ebb2[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:56,229 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:56,229 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:56,230 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:56,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:56,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:56,230 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:56,230 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:56,230 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:56,230 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,230 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:56,232 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,232 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:56,232 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:56,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:56,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:56,232 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:56,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:56,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:56,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:56,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:56,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:56,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:56,235 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:56,235 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:56,236 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:56,236 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:56,236 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:56,236 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:56,237 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:56,237 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:56,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:56,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:56,237 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:56,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:56,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:56,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:56,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:56,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:56,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:56,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:56,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:56,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:56,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:56,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,239 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:56,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:56,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,240 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,243 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:56,243 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:56,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:56,243 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:56,243 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:56,243 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:56,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:56,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:56,243 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:56,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:56,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:56,244 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,244 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:56,244 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:56,244 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:56,244 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:56,244 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:56,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 2764d7f8155e1e5b0d641eebe063fa00 2023-04-09 09:55:56,247 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2764d7f8155e1e5b0d641eebe063fa00 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-09 09:55:56,247 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=A 2023-04-09 09:55:56,248 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:56,248 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=B 2023-04-09 09:55:56,248 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:56,248 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=C 2023-04-09 09:55:56,248 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:56,338 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681034156311, max timestamp=1681034156336Num uniques 859; 2023-04-09 09:55:56,338 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681034156311, max timestamp=1681034156336Num uniques 859; 2023-04-09 09:55:56,338 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-09 09:55:56,338 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681034156311, max timestamp=1681034156336Num uniques 859; 2023-04-09 09:55:56,338 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-09 09:55:56,338 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-09 09:55:56,344 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:56,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:56,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:56,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:56,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:56,347 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:56,347 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:56,347 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:56,347 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:56,348 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:56,348 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:56,349 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,349 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:56,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:56,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:56,350 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,350 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:56,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:56,350 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:56,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:56,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:56,351 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:56,351 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:56,351 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:56,351 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:56,351 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:56,351 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:56,351 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:56,351 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:56,352 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:56,352 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:56,352 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. as already flushing 2023-04-09 09:55:56,352 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:56,352 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:56,352 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:56,352 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,352 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:56,352 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:56,355 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,355 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:56,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:56,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:56,355 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:56,355 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:56,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:56,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:56,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:56,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:56,357 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:56,357 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ae57142[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:56,357 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:56,357 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:56,359 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:56,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:56,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:56,359 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:56,359 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:56,359 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:56,359 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,359 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:56,361 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,361 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:56,361 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:56,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:56,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:56,361 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:56,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:56,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:56,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:56,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:56,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:56,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:56,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:56,363 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:56,363 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:56,363 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:56,363 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:56,363 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:56,364 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:56,364 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:56,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:56,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:56,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:56,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:56,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:56,365 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:56,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:56,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:56,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:56,365 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:56,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:56,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:56,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:56,366 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:56,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:56,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,368 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,368 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:56,368 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:56,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:56,368 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:56,368 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:56,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:56,368 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:56,368 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:56,368 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:56,368 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:56,369 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:56,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:56,369 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:56,369 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:56,369 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:56,369 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:56,369 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:56,440 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:56,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 312 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59344 deadline: 1681034216440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,469 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:56,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:56,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:56,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:56,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:56,473 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:56,473 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:56,473 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:56,473 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:56,476 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:56,476 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:56,479 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:56,479 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:56,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:56,479 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,479 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:56,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:56,479 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:56,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:56,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:56,480 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:56,484 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:56,484 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:56,484 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:56,484 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:56,484 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:56,485 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:56,485 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:56,485 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:56,485 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:56,485 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. as already flushing 2023-04-09 09:55:56,485 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:56,485 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:56,485 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:56,485 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,485 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:56,485 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:56,487 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,487 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:56,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:56,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:56,487 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:56,487 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:56,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:56,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:56,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:56,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:56,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:56,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@382e2697[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:56,488 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:56,489 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:56,489 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:56,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:56,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:56,490 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:56,490 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:56,490 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:56,490 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,490 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:56,491 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:56,491 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:56,491 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:56,491 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:56,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:56,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:56,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:56,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:56,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:56,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:56,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:56,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:56,494 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:56,494 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:56,494 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:56,494 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:56,494 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:56,497 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:56,497 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:56,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:56,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:56,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:56,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:56,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:56,497 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:56,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:56,498 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:56,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:56,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:56,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:56,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:56,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:56,499 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:56,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:56,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,501 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,501 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:56,501 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:56,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:56,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:56,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:56,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:56,501 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:56,501 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:56,501 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,502 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:56,502 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:56,501 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:56,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:56,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:56,502 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:56,501 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:56,502 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:56,502 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:56,545 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:56,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 314 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59344 deadline: 1681034216541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,602 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:56,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:56,604 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:56,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:56,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:56,605 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:56,605 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:56,605 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:56,605 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:56,606 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:56,606 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:56,607 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:56,607 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:56,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:56,607 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,607 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:56,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:56,608 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:56,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:56,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:56,608 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:56,609 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:56,609 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:56,609 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:56,609 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:56,609 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:56,609 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:56,609 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:56,609 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:56,609 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:56,609 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. as already flushing 2023-04-09 09:55:56,609 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:56,609 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:56,609 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:56,609 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,609 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:56,609 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:56,610 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,610 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:56,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:56,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:56,611 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:56,611 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:56,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:56,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:56,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:56,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:56,612 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:56,612 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1dd118c5[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:56,612 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:56,612 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:56,615 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:56,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:56,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:56,615 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:56,615 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:56,615 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:56,615 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,615 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:56,616 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:56,616 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:56,616 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,616 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:56,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:56,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:56,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:56,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:56,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:56,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:56,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:56,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:56,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:56,619 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:56,619 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:56,619 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:56,619 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:56,619 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:56,620 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:56,620 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:56,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:56,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:56,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:56,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:56,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:56,620 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:56,620 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:56,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:56,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:56,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:56,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:56,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:56,623 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:56,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:56,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:56,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,624 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,624 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:56,625 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:56,625 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:56,625 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:56,625 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,625 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:56,625 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:56,625 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:56,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:56,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:56,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:56,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:56,625 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:56,625 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:56,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:56,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:56,625 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:56,625 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:56,654 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=830 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/9ded6dad37734164b98a733f2571ff6c 2023-04-09 09:55:56,725 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:56,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:56,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:56,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:56,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:56,729 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:56,729 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:56,729 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:56,729 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:56,729 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:56,729 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:56,731 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,731 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:56,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:56,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:56,732 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,732 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:56,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:56,732 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:56,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:56,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:56,732 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:56,732 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:56,733 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:56,733 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:56,733 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:56,733 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:56,733 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:56,733 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:56,733 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:56,733 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:56,733 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. as already flushing 2023-04-09 09:55:56,733 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:56,733 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:56,733 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:56,733 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,733 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:56,733 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:56,734 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:56,735 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:56,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:56,735 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:56,735 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:56,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:56,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:56,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:56,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:56,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:56,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e3cab6[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:56,736 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:56,736 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:56,737 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:56,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:56,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:56,737 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:56,737 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:56,737 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:56,737 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,737 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:56,738 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,739 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:56,739 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:56,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:56,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:56,739 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:56,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:56,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:56,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:56,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:56,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:56,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:56,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:56,741 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:56,741 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:56,741 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:56,741 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:56,741 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:56,746 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:56,746 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:56,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:56,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:56,746 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:56,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:56,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:56,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:56,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:56,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:56,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:56,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:56,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:56,748 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:56,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:56,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,749 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:56,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:56,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 316 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59344 deadline: 1681034216746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,749 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:56,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,751 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:56,751 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:56,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:56,751 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:56,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:56,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:56,751 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:56,751 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:56,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:56,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:56,752 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:56,752 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:56,752 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:56,752 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,752 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:56,752 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:56,752 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:56,852 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:56,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:56,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:56,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:56,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:56,855 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:56,855 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:56,855 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:56,855 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:56,856 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:56,856 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:56,857 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,857 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:56,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:56,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:56,857 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,857 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:56,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:56,858 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:56,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:56,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:56,858 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:56,858 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:56,858 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:56,858 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:56,858 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:56,858 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:56,859 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:56,859 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:56,859 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:56,859 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:56,859 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. as already flushing 2023-04-09 09:55:56,859 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:56,859 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:56,859 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:56,859 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,859 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:56,859 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:56,862 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,862 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:56,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:56,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:56,862 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:56,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:56,862 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:56,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:56,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:56,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:56,864 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:56,864 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40dd429[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:56,864 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:56,864 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:56,865 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:56,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:56,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:56,865 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:56,865 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,865 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:56,865 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:56,865 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:56,866 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:56,866 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:56,866 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,866 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:56,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:56,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:56,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:56,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:56,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:56,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:56,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:56,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:56,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:56,869 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:56,869 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:56,869 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:56,869 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:56,869 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:56,870 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:56,870 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:56,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:56,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:56,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:56,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:56,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:56,870 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:56,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:56,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:56,871 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:56,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:56,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:56,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:56,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:56,872 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:56,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:56,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,879 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:56,879 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:56,879 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:56,879 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:56,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:56,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:56,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:56,879 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:56,879 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:56,879 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:56,879 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,879 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:56,879 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:56,879 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:56,879 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:56,879 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:56,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:56,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:56,980 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:56,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:56,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:56,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:56,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:56,983 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:56,983 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:56,983 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:56,983 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:56,983 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:56,983 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:56,984 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:56,984 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:56,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:56,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:56,985 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,985 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:56,985 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:56,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:56,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:56,986 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:56,986 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:56,986 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:56,986 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:56,986 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:56,986 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:56,986 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:56,986 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:56,986 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:56,986 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:56,986 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. as already flushing 2023-04-09 09:55:56,986 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:56,986 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:56,987 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:56,987 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,987 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:56,987 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:56,988 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,988 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:56,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:56,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:56,988 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:56,988 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:56,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:56,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:56,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:56,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:56,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:56,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d5e50a1[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:56,990 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:56,990 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:56,991 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:56,991 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,991 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:56,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:56,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:56,992 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:56,992 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:56,992 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:56,994 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:56,994 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:56,994 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,994 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:56,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:56,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:56,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:56,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:56,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:56,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:56,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:56,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:56,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:56,997 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:56,997 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:56,997 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:56,997 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:56,997 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:56,998 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:56,998 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:56,998 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:56,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:56,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:56,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:56,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:56,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:56,998 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:56,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:56,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:56,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:56,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:56,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:56,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:56,999 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:57,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:57,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:57,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:57,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:57,004 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:57,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:57,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:57,005 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:57,005 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:57,005 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:57,005 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:57,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:57,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:57,006 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:57,006 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:57,006 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:57,006 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:57,006 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:57,006 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:57,006 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:57,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:57,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:57,006 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:57,058 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:57,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 318 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59344 deadline: 1681034217052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:57,063 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=830 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/5f3e4d7a22c14140a29263d83d34dc06 2023-04-09 09:55:57,072 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=830 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/3ec990e42a224608975e57d2b143754a 2023-04-09 09:55:57,075 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/9ded6dad37734164b98a733f2571ff6c as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/9ded6dad37734164b98a733f2571ff6c 2023-04-09 09:55:57,078 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/9ded6dad37734164b98a733f2571ff6c, entries=200, sequenceid=830, filesize=14.3 K 2023-04-09 09:55:57,078 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/5f3e4d7a22c14140a29263d83d34dc06 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/5f3e4d7a22c14140a29263d83d34dc06 2023-04-09 09:55:57,081 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/5f3e4d7a22c14140a29263d83d34dc06, entries=150, sequenceid=830, filesize=11.9 K 2023-04-09 09:55:57,082 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/3ec990e42a224608975e57d2b143754a as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/3ec990e42a224608975e57d2b143754a 2023-04-09 09:55:57,085 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/3ec990e42a224608975e57d2b143754a, entries=150, sequenceid=830, filesize=11.9 K 2023-04-09 09:55:57,085 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 2764d7f8155e1e5b0d641eebe063fa00 in 838ms, sequenceid=830, compaction requested=true 2023-04-09 09:55:57,085 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2764d7f8155e1e5b0d641eebe063fa00: 2023-04-09 09:55:57,085 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:55:57,085 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-09 09:55:57,086 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-09 09:55:57,085 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:55:57,086 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:55:57,087 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 64562 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-09 09:55:57,087 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1912): 2764d7f8155e1e5b0d641eebe063fa00/A is initiating minor compaction (all files) 2023-04-09 09:55:57,087 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 62122 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-09 09:55:57,087 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2764d7f8155e1e5b0d641eebe063fa00/A in TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:57,087 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): 2764d7f8155e1e5b0d641eebe063fa00/B is initiating minor compaction (all files) 2023-04-09 09:55:57,087 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/accb65ed4e9348c2ac901af52ed54730, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/41534cc5172b4069b2743301a4081526, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/0f8e534b382a44f8b3393433187502dc, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/48abc2ed111c4b50b033176785aa21cd, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/9ded6dad37734164b98a733f2571ff6c] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp, totalSize=63.0 K 2023-04-09 09:55:57,087 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2764d7f8155e1e5b0d641eebe063fa00/B in TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:57,087 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/9081f967b0f849fa82ab4baa2d3e857c, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/18dce3b200ac48c796c51cc1d7421fc4, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/c4659bdfb8844ef2a3da8aa1d8f86283, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/c53eb5057bad4b9c8cc79653fa633d01, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/5f3e4d7a22c14140a29263d83d34dc06] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp, totalSize=60.7 K 2023-04-09 09:55:57,087 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting accb65ed4e9348c2ac901af52ed54730, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=733, earliestPutTs=1681034153324 2023-04-09 09:55:57,088 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 9081f967b0f849fa82ab4baa2d3e857c, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=733, earliestPutTs=1681034153324 2023-04-09 09:55:57,088 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 41534cc5172b4069b2743301a4081526, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=772, earliestPutTs=1681034153498 2023-04-09 09:55:57,088 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 18dce3b200ac48c796c51cc1d7421fc4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=772, earliestPutTs=1681034153498 2023-04-09 09:55:57,088 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 0f8e534b382a44f8b3393433187502dc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=783, earliestPutTs=1681034154244 2023-04-09 09:55:57,088 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting c4659bdfb8844ef2a3da8aa1d8f86283, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=783, earliestPutTs=1681034154244 2023-04-09 09:55:57,088 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 48abc2ed111c4b50b033176785aa21cd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=820, earliestPutTs=1681034155538 2023-04-09 09:55:57,089 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting c53eb5057bad4b9c8cc79653fa633d01, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=820, earliestPutTs=1681034155538 2023-04-09 09:55:57,089 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 9ded6dad37734164b98a733f2571ff6c, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=830, earliestPutTs=1681034156232 2023-04-09 09:55:57,089 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 5f3e4d7a22c14140a29263d83d34dc06, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=830, earliestPutTs=1681034156236 2023-04-09 09:55:57,098 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): 2764d7f8155e1e5b0d641eebe063fa00#B#compaction#538 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:55:57,098 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2764d7f8155e1e5b0d641eebe063fa00#A#compaction#537 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:55:57,105 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:57,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:57,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:57,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:57,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:57,108 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:57,108 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:57,108 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:57,108 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:57,108 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:57,108 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:57,109 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:57,109 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:57,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:57,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:57,110 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:57,110 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:57,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:57,110 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:57,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:57,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:57,110 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:57,111 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:57,111 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:57,111 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:57,111 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:57,111 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:57,111 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:57,111 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:57,111 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:57,111 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:57,111 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2764d7f8155e1e5b0d641eebe063fa00 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-09 09:55:57,112 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=A 2023-04-09 09:55:57,112 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:55:57,112 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=B 2023-04-09 09:55:57,112 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:55:57,112 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=C 2023-04-09 09:55:57,112 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:55:57,506 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/3a43eff56fdb4ef58022e3aa86724880 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/3a43eff56fdb4ef58022e3aa86724880 2023-04-09 09:55:57,510 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/0ba49e50c10b4e2b9068f37b13629458 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/0ba49e50c10b4e2b9068f37b13629458 2023-04-09 09:55:57,511 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 2764d7f8155e1e5b0d641eebe063fa00/B of 2764d7f8155e1e5b0d641eebe063fa00 into 3a43eff56fdb4ef58022e3aa86724880(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:55:57,511 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2764d7f8155e1e5b0d641eebe063fa00: 2023-04-09 09:55:57,511 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00., storeName=2764d7f8155e1e5b0d641eebe063fa00/B, priority=11, startTime=1681034157085; duration=0sec 2023-04-09 09:55:57,511 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:55:57,511 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-09 09:55:57,513 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 62122 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-09 09:55:57,513 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): 2764d7f8155e1e5b0d641eebe063fa00/C is initiating minor compaction (all files) 2023-04-09 09:55:57,513 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2764d7f8155e1e5b0d641eebe063fa00/C in TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:57,513 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/51a816e2764b4fba8ef0a6119b45e0a6, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/5419ea9f45764d09b29444986d77f596, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/56942368a0e648be9beef5ef67df0c07, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/155bbfdf6f3643caaedd0d4579ed5663, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/3ec990e42a224608975e57d2b143754a] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp, totalSize=60.7 K 2023-04-09 09:55:57,514 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 51a816e2764b4fba8ef0a6119b45e0a6, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=733, earliestPutTs=1681034153324 2023-04-09 09:55:57,514 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 5419ea9f45764d09b29444986d77f596, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=772, earliestPutTs=1681034153498 2023-04-09 09:55:57,514 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 2764d7f8155e1e5b0d641eebe063fa00/A of 2764d7f8155e1e5b0d641eebe063fa00 into 0ba49e50c10b4e2b9068f37b13629458(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:55:57,514 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2764d7f8155e1e5b0d641eebe063fa00: 2023-04-09 09:55:57,514 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00., storeName=2764d7f8155e1e5b0d641eebe063fa00/A, priority=11, startTime=1681034157085; duration=0sec 2023-04-09 09:55:57,514 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 56942368a0e648be9beef5ef67df0c07, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=783, earliestPutTs=1681034154244 2023-04-09 09:55:57,514 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:55:57,515 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 155bbfdf6f3643caaedd0d4579ed5663, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=820, earliestPutTs=1681034155538 2023-04-09 09:55:57,515 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 3ec990e42a224608975e57d2b143754a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=830, earliestPutTs=1681034156236 2023-04-09 09:55:57,523 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): 2764d7f8155e1e5b0d641eebe063fa00#C#compaction#540 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:55:57,524 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=866 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/29bbfaee8bf9469fa5adc9a0bef3f28d 2023-04-09 09:55:57,539 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/6d5a94f4c2224455939b9f0740fdca44 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/6d5a94f4c2224455939b9f0740fdca44 2023-04-09 09:55:57,544 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=866 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/14996b8cd81b49178814a994335d049b 2023-04-09 09:55:57,545 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 2764d7f8155e1e5b0d641eebe063fa00/C of 2764d7f8155e1e5b0d641eebe063fa00 into 6d5a94f4c2224455939b9f0740fdca44(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:55:57,545 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2764d7f8155e1e5b0d641eebe063fa00: 2023-04-09 09:55:57,545 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00., storeName=2764d7f8155e1e5b0d641eebe063fa00/C, priority=11, startTime=1681034157086; duration=0sec 2023-04-09 09:55:57,545 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:55:57,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 2764d7f8155e1e5b0d641eebe063fa00 2023-04-09 09:55:57,562 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. as already flushing 2023-04-09 09:55:57,595 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:57,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 327 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59344 deadline: 1681034217591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:57,675 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:57,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59328 deadline: 1681034217673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:57,679 DEBUG [Thread-3182] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4158 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00., hostname=jenkins-hbase4.apache.org,42467,1681034019544, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-09 09:55:57,682 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:57,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59302 deadline: 1681034217676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:57,683 DEBUG [Thread-3174] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4177 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00., hostname=jenkins-hbase4.apache.org,42467,1681034019544, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-09 09:55:57,697 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:57,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 329 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59344 deadline: 1681034217696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:57,902 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:57,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 331 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59344 deadline: 1681034217899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:57,953 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=866 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/91c73064316c463c93922d7b9599746f 2023-04-09 09:55:57,956 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/29bbfaee8bf9469fa5adc9a0bef3f28d as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/29bbfaee8bf9469fa5adc9a0bef3f28d 2023-04-09 09:55:57,959 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/29bbfaee8bf9469fa5adc9a0bef3f28d, entries=150, sequenceid=866, filesize=11.9 K 2023-04-09 09:55:57,960 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/14996b8cd81b49178814a994335d049b as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/14996b8cd81b49178814a994335d049b 2023-04-09 09:55:57,963 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/14996b8cd81b49178814a994335d049b, entries=150, sequenceid=866, filesize=11.9 K 2023-04-09 09:55:57,963 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/91c73064316c463c93922d7b9599746f as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/91c73064316c463c93922d7b9599746f 2023-04-09 09:55:57,966 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/91c73064316c463c93922d7b9599746f, entries=150, sequenceid=866, filesize=11.9 K 2023-04-09 09:55:57,967 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 2764d7f8155e1e5b0d641eebe063fa00 in 856ms, sequenceid=866, compaction requested=false 2023-04-09 09:55:57,967 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2764d7f8155e1e5b0d641eebe063fa00: 2023-04-09 09:55:57,967 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:57,967 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:57,967 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:57,967 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:57,967 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:57,967 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:57,968 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:57,968 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:57,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:57,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:57,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:57,969 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:57,969 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:57,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:57,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:57,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:57,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:57,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:57,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:57,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58dc7eff[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:57,970 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:57,970 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:57,972 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:57,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:57,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:57,972 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:57,972 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:57,972 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:57,972 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:57,972 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:57,973 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:57,973 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:57,973 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:57,973 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:57,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:57,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:57,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:57,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:57,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:57,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:57,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:57,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:57,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:57,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:57,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:57,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:57,976 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:57,976 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:57,976 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:57,976 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:57,976 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:57,977 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:57,977 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:57,977 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:57,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:57,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:57,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:57,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:57,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:57,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:57,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:57,978 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:57,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:57,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:57,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:57,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:57,979 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:57,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:57,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:57,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:57,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:57,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:57,985 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:57,985 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:57,985 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:57,985 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:57,985 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:57,985 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:57,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:57,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:57,985 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:57,985 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:57,985 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:55:57,985 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:57,985 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:57,985 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:57,985 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:57,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:57,985 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:57,986 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:57,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:58,086 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:58,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:58,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:55:58,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:55:58,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:55:58,089 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:55:58,089 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:58,089 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:55:58,089 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:55:58,090 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:58,090 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:58,091 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:58,091 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:58,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:58,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:58,091 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:58,091 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:55:58,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:58,091 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:58,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:55:58,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:58,092 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:55:58,092 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:55:58,092 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:55:58,092 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:55:58,092 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:55:58,092 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:55:58,092 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:55:58,092 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:55:58,092 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:58,092 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. started... 2023-04-09 09:55:58,093 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2764d7f8155e1e5b0d641eebe063fa00 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-09 09:55:58,093 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=A 2023-04-09 09:55:58,093 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:58,093 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=B 2023-04-09 09:55:58,093 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:58,093 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=C 2023-04-09 09:55:58,093 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:55:58,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 2764d7f8155e1e5b0d641eebe063fa00 2023-04-09 09:55:58,215 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. as already flushing 2023-04-09 09:55:58,295 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034158280, max timestamp=1681034158294Num uniques 859; 2023-04-09 09:55:58,295 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:55:58,296 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034158280, max timestamp=1681034158294Num uniques 859; 2023-04-09 09:55:58,296 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034158280, max timestamp=1681034158294Num uniques 859; 2023-04-09 09:55:58,296 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:55:58,296 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:55:58,348 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:58,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59316 deadline: 1681034218346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:58,357 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:58,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 360 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59344 deadline: 1681034218347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:58,357 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:58,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59348 deadline: 1681034218348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:58,459 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:58,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59316 deadline: 1681034218449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:58,460 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:58,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 362 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59344 deadline: 1681034218458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:58,460 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:58,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59348 deadline: 1681034218458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:58,500 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=879 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/0b9a65f09e9a40a6b94e4aafdb985c56 2023-04-09 09:55:58,662 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:58,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59316 deadline: 1681034218661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:58,662 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:58,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59348 deadline: 1681034218661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:58,662 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:58,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 364 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59344 deadline: 1681034218662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:58,913 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=879 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/cb44afe2b20341f2b9ab7d38f3f12dfb 2023-04-09 09:55:58,965 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:58,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59348 deadline: 1681034218963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:58,965 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:58,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 366 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59344 deadline: 1681034218963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:58,969 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:55:58,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59316 deadline: 1681034218964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:59,181 DEBUG [Thread-3189] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x78592df0 to 127.0.0.1:57816 2023-04-09 09:55:59,182 DEBUG [Thread-3189] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-09 09:55:59,185 DEBUG [Thread-3185] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x70e2be04 to 127.0.0.1:57816 2023-04-09 09:55:59,185 DEBUG [Thread-3185] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-09 09:55:59,186 DEBUG [Thread-3193] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x680e773e to 127.0.0.1:57816 2023-04-09 09:55:59,186 DEBUG [Thread-3193] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-09 09:55:59,188 DEBUG [Thread-3187] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x067ea274 to 127.0.0.1:57816 2023-04-09 09:55:59,188 DEBUG [Thread-3187] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-09 09:55:59,191 DEBUG [Thread-3191] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x659c68ab to 127.0.0.1:57816 2023-04-09 09:55:59,191 DEBUG [Thread-3191] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-09 09:55:59,326 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=879 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/68fedc8964554ff3954f890d31704664 2023-04-09 09:55:59,329 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/0b9a65f09e9a40a6b94e4aafdb985c56 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/0b9a65f09e9a40a6b94e4aafdb985c56 2023-04-09 09:55:59,333 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/0b9a65f09e9a40a6b94e4aafdb985c56, entries=150, sequenceid=879, filesize=11.9 K 2023-04-09 09:55:59,334 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/cb44afe2b20341f2b9ab7d38f3f12dfb as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/cb44afe2b20341f2b9ab7d38f3f12dfb 2023-04-09 09:55:59,338 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/cb44afe2b20341f2b9ab7d38f3f12dfb, entries=150, sequenceid=879, filesize=11.9 K 2023-04-09 09:55:59,338 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/68fedc8964554ff3954f890d31704664 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/68fedc8964554ff3954f890d31704664 2023-04-09 09:55:59,341 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/68fedc8964554ff3954f890d31704664, entries=150, sequenceid=879, filesize=11.9 K 2023-04-09 09:55:59,342 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 2764d7f8155e1e5b0d641eebe063fa00 in 1249ms, sequenceid=879, compaction requested=true 2023-04-09 09:55:59,342 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2764d7f8155e1e5b0d641eebe063fa00: 2023-04-09 09:55:59,342 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:55:59,342 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:55:59,342 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:55:59,342 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:59,342 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:55:59,343 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:55:59,344 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:59,344 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:59,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:59,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:59,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:59,344 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:59,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:59,344 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:55:59,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:59,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:59,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:59,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:59,345 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:55:59,345 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43f1a928[Count = 0] remaining members to acquire global barrier 2023-04-09 09:55:59,345 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:55:59,346 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:59,348 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:59,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:59,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:59,348 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:55:59,348 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:55:59,348 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:55:59,348 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:59,348 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:55:59,349 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:59,349 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:55:59,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:59,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:59,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:59,349 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:59,349 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:55:59,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:59,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:59,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:59,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:59,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:59,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:59,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:59,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:55:59,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:55:59,351 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:55:59,351 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:55:59,351 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:55:59,351 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:55:59,351 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:55:59,353 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:59,353 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:59,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:59,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:59,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:59,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:55:59,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:55:59,353 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:59,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:59,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:59,353 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:59,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:55:59,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:59,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:59,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:55:59,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:59,354 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:59,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:59,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:55:59,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:55:59,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:59,356 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:59,356 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:55:59,356 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:59,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:55:59,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:55:59,356 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:55:59,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:55:59,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:55:59,356 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:55:59,356 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:55:59,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:55:59,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:55:59,356 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:55:59,356 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:59,356 DEBUG [Thread-3184] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:55:59,356 DEBUG [Thread-3184] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:55:59,356 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:55:59,356 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:55:59,456 DEBUG [Thread-3184] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:55:59,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:55:59,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on 2764d7f8155e1e5b0d641eebe063fa00 2023-04-09 09:55:59,468 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2764d7f8155e1e5b0d641eebe063fa00 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-04-09 09:55:59,468 DEBUG [Thread-3178] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x052203cb to 127.0.0.1:57816 2023-04-09 09:55:59,468 DEBUG [Thread-3178] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-09 09:55:59,468 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=A 2023-04-09 09:55:59,468 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-09 09:55:59,468 DEBUG [Thread-3180] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x420f945a to 127.0.0.1:57816 2023-04-09 09:55:59,468 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=B 2023-04-09 09:55:59,468 DEBUG [Thread-3180] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-09 09:55:59,468 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-09 09:55:59,468 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=C 2023-04-09 09:55:59,468 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-09 09:55:59,474 DEBUG [Thread-3176] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x15880ffb to 127.0.0.1:57816 2023-04-09 09:55:59,474 DEBUG [Thread-3176] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-09 09:55:59,474 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=918 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/d17a517883fb48208514cb1795e1ab78 2023-04-09 09:55:59,882 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=918 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/a7c07597f6b04d0594e0f3d90c3dce8d 2023-04-09 09:56:00,292 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=918 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/f103666abaef4742966d23362b62092f 2023-04-09 09:56:00,295 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/d17a517883fb48208514cb1795e1ab78 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/d17a517883fb48208514cb1795e1ab78 2023-04-09 09:56:00,297 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/d17a517883fb48208514cb1795e1ab78, entries=150, sequenceid=918, filesize=11.9 K 2023-04-09 09:56:00,298 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/a7c07597f6b04d0594e0f3d90c3dce8d as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/a7c07597f6b04d0594e0f3d90c3dce8d 2023-04-09 09:56:00,301 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/a7c07597f6b04d0594e0f3d90c3dce8d, entries=150, sequenceid=918, filesize=11.9 K 2023-04-09 09:56:00,301 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/f103666abaef4742966d23362b62092f as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/f103666abaef4742966d23362b62092f 2023-04-09 09:56:00,304 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/f103666abaef4742966d23362b62092f, entries=150, sequenceid=918, filesize=11.9 K 2023-04-09 09:56:00,304 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~425.48 KB/435696, currentSize=6.71 KB/6870 for 2764d7f8155e1e5b0d641eebe063fa00 in 836ms, sequenceid=918, compaction requested=true 2023-04-09 09:56:00,304 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2764d7f8155e1e5b0d641eebe063fa00: 2023-04-09 09:56:00,305 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:56:00,305 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-09 09:56:00,305 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:56:00,305 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-09 09:56:00,305 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:56:00,306 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 50106 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-09 09:56:00,306 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 50106 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-09 09:56:00,306 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1912): 2764d7f8155e1e5b0d641eebe063fa00/A is initiating minor compaction (all files) 2023-04-09 09:56:00,306 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): 2764d7f8155e1e5b0d641eebe063fa00/B is initiating minor compaction (all files) 2023-04-09 09:56:00,306 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2764d7f8155e1e5b0d641eebe063fa00/A in TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:56:00,306 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2764d7f8155e1e5b0d641eebe063fa00/B in TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:56:00,306 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/0ba49e50c10b4e2b9068f37b13629458, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/29bbfaee8bf9469fa5adc9a0bef3f28d, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/0b9a65f09e9a40a6b94e4aafdb985c56, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/d17a517883fb48208514cb1795e1ab78] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp, totalSize=48.9 K 2023-04-09 09:56:00,306 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/3a43eff56fdb4ef58022e3aa86724880, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/14996b8cd81b49178814a994335d049b, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/cb44afe2b20341f2b9ab7d38f3f12dfb, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/a7c07597f6b04d0594e0f3d90c3dce8d] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp, totalSize=48.9 K 2023-04-09 09:56:00,306 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 0ba49e50c10b4e2b9068f37b13629458, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=830, earliestPutTs=1681034156236 2023-04-09 09:56:00,306 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 3a43eff56fdb4ef58022e3aa86724880, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=830, earliestPutTs=1681034156236 2023-04-09 09:56:00,306 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 29bbfaee8bf9469fa5adc9a0bef3f28d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=866, earliestPutTs=1681034156422 2023-04-09 09:56:00,307 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 14996b8cd81b49178814a994335d049b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=866, earliestPutTs=1681034156422 2023-04-09 09:56:00,307 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 0b9a65f09e9a40a6b94e4aafdb985c56, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=879, earliestPutTs=1681034157563 2023-04-09 09:56:00,307 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting cb44afe2b20341f2b9ab7d38f3f12dfb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=879, earliestPutTs=1681034157563 2023-04-09 09:56:00,307 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting d17a517883fb48208514cb1795e1ab78, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=918, earliestPutTs=1681034158328 2023-04-09 09:56:00,307 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting a7c07597f6b04d0594e0f3d90c3dce8d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=918, earliestPutTs=1681034158328 2023-04-09 09:56:00,314 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2764d7f8155e1e5b0d641eebe063fa00#A#compaction#549 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:56:00,314 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): 2764d7f8155e1e5b0d641eebe063fa00#B#compaction#550 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:56:00,320 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/c6171ba2a70549a3ab47363a40916b24 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/c6171ba2a70549a3ab47363a40916b24 2023-04-09 09:56:00,323 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2764d7f8155e1e5b0d641eebe063fa00/B of 2764d7f8155e1e5b0d641eebe063fa00 into c6171ba2a70549a3ab47363a40916b24(size=13.4 K), total size for store is 13.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:56:00,323 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2764d7f8155e1e5b0d641eebe063fa00: 2023-04-09 09:56:00,323 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00., storeName=2764d7f8155e1e5b0d641eebe063fa00/B, priority=12, startTime=1681034160305; duration=0sec 2023-04-09 09:56:00,324 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:56:00,324 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-09 09:56:00,324 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 50106 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-09 09:56:00,324 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): 2764d7f8155e1e5b0d641eebe063fa00/C is initiating minor compaction (all files) 2023-04-09 09:56:00,325 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2764d7f8155e1e5b0d641eebe063fa00/C in TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:56:00,325 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/6d5a94f4c2224455939b9f0740fdca44, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/91c73064316c463c93922d7b9599746f, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/68fedc8964554ff3954f890d31704664, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/f103666abaef4742966d23362b62092f] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp, totalSize=48.9 K 2023-04-09 09:56:00,325 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 6d5a94f4c2224455939b9f0740fdca44, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=830, earliestPutTs=1681034156236 2023-04-09 09:56:00,325 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 91c73064316c463c93922d7b9599746f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=866, earliestPutTs=1681034156422 2023-04-09 09:56:00,325 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 68fedc8964554ff3954f890d31704664, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=879, earliestPutTs=1681034157563 2023-04-09 09:56:00,326 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting f103666abaef4742966d23362b62092f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=918, earliestPutTs=1681034158328 2023-04-09 09:56:00,331 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): 2764d7f8155e1e5b0d641eebe063fa00#C#compaction#551 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:56:00,721 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/4f365d8c365f49ddb0ff5292287f41a9 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/4f365d8c365f49ddb0ff5292287f41a9 2023-04-09 09:56:00,724 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2764d7f8155e1e5b0d641eebe063fa00/A of 2764d7f8155e1e5b0d641eebe063fa00 into 4f365d8c365f49ddb0ff5292287f41a9(size=13.4 K), total size for store is 13.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:56:00,724 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2764d7f8155e1e5b0d641eebe063fa00: 2023-04-09 09:56:00,724 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00., storeName=2764d7f8155e1e5b0d641eebe063fa00/A, priority=12, startTime=1681034160304; duration=0sec 2023-04-09 09:56:00,724 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:56:00,737 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/f62a9f9514684bc2aadec973fa854c42 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/f62a9f9514684bc2aadec973fa854c42 2023-04-09 09:56:00,740 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2764d7f8155e1e5b0d641eebe063fa00/C of 2764d7f8155e1e5b0d641eebe063fa00 into f62a9f9514684bc2aadec973fa854c42(size=13.4 K), total size for store is 13.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:56:00,740 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2764d7f8155e1e5b0d641eebe063fa00: 2023-04-09 09:56:00,740 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00., storeName=2764d7f8155e1e5b0d641eebe063fa00/C, priority=12, startTime=1681034160305; duration=0sec 2023-04-09 09:56:00,740 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:56:01,706 DEBUG [Thread-3182] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x51b8df2a to 127.0.0.1:57816 2023-04-09 09:56:01,706 DEBUG [Thread-3182] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-09 09:56:01,716 DEBUG [Thread-3174] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x37705401 to 127.0.0.1:57816 2023-04-09 09:56:01,716 DEBUG [Thread-3174] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-09 09:56:01,716 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-09 09:56:01,717 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(394): wrote 114 2023-04-09 09:56:01,717 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(394): wrote 158 2023-04-09 09:56:01,717 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(394): wrote 247 2023-04-09 09:56:01,717 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(394): wrote 160 2023-04-09 09:56:01,717 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(394): wrote 108 2023-04-09 09:56:01,717 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-09 09:56:01,717 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-09 09:56:01,717 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(402): scanned 1903 2023-04-09 09:56:01,717 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(403): verified 5709 rows 2023-04-09 09:56:01,717 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(402): scanned 1904 2023-04-09 09:56:01,717 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(403): verified 5712 rows 2023-04-09 09:56:01,717 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(402): scanned 1917 2023-04-09 09:56:01,717 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(403): verified 5751 rows 2023-04-09 09:56:01,717 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(402): scanned 1895 2023-04-09 09:56:01,717 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(403): verified 5685 rows 2023-04-09 09:56:01,717 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(402): scanned 1911 2023-04-09 09:56:01,717 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(403): verified 5730 rows 2023-04-09 09:56:01,717 INFO [Listener at localhost/44563] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-09 09:56:01,717 DEBUG [Listener at localhost/44563] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2a987907 to 127.0.0.1:57816 2023-04-09 09:56:01,717 DEBUG [Listener at localhost/44563] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-09 09:56:01,721 INFO [Listener at localhost/44563] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-09 09:56:01,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-09 09:56:01,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure2.ProcedureExecutor(1029): Stored pid=50, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-09 09:56:01,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-09 09:56:01,725 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681034161725"}]},"ts":"1681034161725"} 2023-04-09 09:56:01,726 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-09 09:56:01,727 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-09 09:56:01,727 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2764d7f8155e1e5b0d641eebe063fa00, UNASSIGN}] 2023-04-09 09:56:01,729 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2764d7f8155e1e5b0d641eebe063fa00, UNASSIGN 2023-04-09 09:56:01,729 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=2764d7f8155e1e5b0d641eebe063fa00, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:01,729 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681034161729"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681034161729"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681034161729"}]},"ts":"1681034161729"} 2023-04-09 09:56:01,730 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=52, ppid=51, state=RUNNABLE; CloseRegionProcedure 2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544}] 2023-04-09 09:56:01,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-09 09:56:01,882 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 2764d7f8155e1e5b0d641eebe063fa00 2023-04-09 09:56:01,882 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 2764d7f8155e1e5b0d641eebe063fa00, disabling compactions & flushes 2023-04-09 09:56:01,882 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:56:01,882 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:56:01,882 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. after waiting 0 ms 2023-04-09 09:56:01,882 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:56:01,882 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 2764d7f8155e1e5b0d641eebe063fa00 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-09 09:56:01,882 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=A 2023-04-09 09:56:01,882 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:56:01,882 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=B 2023-04-09 09:56:01,882 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:56:01,882 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2764d7f8155e1e5b0d641eebe063fa00, store=C 2023-04-09 09:56:01,882 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:56:02,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-09 09:56:02,289 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=927 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/dd4b9c4065e34d65adb673f0404e4bb8 2023-04-09 09:56:02,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-09 09:56:02,696 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=927 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/73f9ea996e5f4f3eb66adde2073599fa 2023-04-09 09:56:02,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-09 09:56:03,104 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=927 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/0731e11aad7b414dae4338943446af08 2023-04-09 09:56:03,107 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/A/dd4b9c4065e34d65adb673f0404e4bb8 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/dd4b9c4065e34d65adb673f0404e4bb8 2023-04-09 09:56:03,110 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/dd4b9c4065e34d65adb673f0404e4bb8, entries=50, sequenceid=927, filesize=7.1 K 2023-04-09 09:56:03,110 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/B/73f9ea996e5f4f3eb66adde2073599fa as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/73f9ea996e5f4f3eb66adde2073599fa 2023-04-09 09:56:03,113 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/73f9ea996e5f4f3eb66adde2073599fa, entries=50, sequenceid=927, filesize=7.1 K 2023-04-09 09:56:03,114 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/.tmp/C/0731e11aad7b414dae4338943446af08 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/0731e11aad7b414dae4338943446af08 2023-04-09 09:56:03,116 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/0731e11aad7b414dae4338943446af08, entries=50, sequenceid=927, filesize=7.1 K 2023-04-09 09:56:03,117 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for 2764d7f8155e1e5b0d641eebe063fa00 in 1235ms, sequenceid=927, compaction requested=false 2023-04-09 09:56:03,141 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/5b7924b6347b4e4a9c093b9ca8268e27, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/8eb41fcb814042ba8b70504eeb903c92, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/8ba5a6c9869d4544ad3f5e08c68a6bb8, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/1481470a0253411aaeea374d1d287e89, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/6b49c1b493f24ec9b68e962c013f676f, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/c69111087cb0438fbb9f69408854a40e, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/7981f64e45b44940a3df8fa4ccc2c45d, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/c9bd2efe5d5c4593925f772659d31c29, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/0d9ea862f2164a0e8d2915e93f3cd501, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/7e3567833c804b3db3968083cab07f45, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/a4e36e418b3e4ff98855327e7780a1cc, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/eee11ccccf93447988aa9d03baedd504, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/0f037ba4104c4f12a69a41ac4f3f34d5, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/b42216dee8dd451080de07cef69f2083, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/66d87d295c4544f68f2138fbae5281ee, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/76aa650003c247beb83aea42f5f03e61, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/b09bec28124a47038a426d9d757d1e2b, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/cddbefe4674e46ef8c9a2cfcb6c2a74e, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/3aadb574b7ea48a39e4a1bc72db41022, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/8483cd68b8b74843ab838e4c89682d13, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/d0174daf06b34a5e8aa2d734b18c5fe4, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/20a34cc1f59d4f27856b60a0687c4d13, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/cdebebce81934e9fa43b2474215ae2ca, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/c3d37b6d3e1046c186779fc4e3c11ff5, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/b1e58e24eef54e16bc7b0a877bf2d790, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/e251a28a28744c259915b1dd81bc6adc, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/230a11401f0441d29e6c8454dbba3b8c, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/7c9e3ced0699439fb67d36440eaef12c, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/6567c7954953486e8877579f00b1bc34, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/ade3157ea4824c0e9be3f1e14883f4f5, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/20e537b474aa44c588ad3d69732713f2, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/c784667685e04953836405e3c674c294, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/de4eec7029aa4036bd620a99d4fec7ce, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/32fa6ba826f146f29b7968d35092b4e3, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/9046d34cd4b84d99b54cfa0b62d7b1c1, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/accb65ed4e9348c2ac901af52ed54730, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/41534cc5172b4069b2743301a4081526, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/0f8e534b382a44f8b3393433187502dc, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/48abc2ed111c4b50b033176785aa21cd, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/9ded6dad37734164b98a733f2571ff6c, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/0ba49e50c10b4e2b9068f37b13629458, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/29bbfaee8bf9469fa5adc9a0bef3f28d, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/0b9a65f09e9a40a6b94e4aafdb985c56, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/d17a517883fb48208514cb1795e1ab78] to archive 2023-04-09 09:56:03,141 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-09 09:56:03,143 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/5b7924b6347b4e4a9c093b9ca8268e27 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/5b7924b6347b4e4a9c093b9ca8268e27 2023-04-09 09:56:03,144 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/8eb41fcb814042ba8b70504eeb903c92 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/8eb41fcb814042ba8b70504eeb903c92 2023-04-09 09:56:03,145 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/8ba5a6c9869d4544ad3f5e08c68a6bb8 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/8ba5a6c9869d4544ad3f5e08c68a6bb8 2023-04-09 09:56:03,145 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/1481470a0253411aaeea374d1d287e89 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/1481470a0253411aaeea374d1d287e89 2023-04-09 09:56:03,146 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/6b49c1b493f24ec9b68e962c013f676f to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/6b49c1b493f24ec9b68e962c013f676f 2023-04-09 09:56:03,147 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/c69111087cb0438fbb9f69408854a40e to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/c69111087cb0438fbb9f69408854a40e 2023-04-09 09:56:03,148 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/7981f64e45b44940a3df8fa4ccc2c45d to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/7981f64e45b44940a3df8fa4ccc2c45d 2023-04-09 09:56:03,149 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/c9bd2efe5d5c4593925f772659d31c29 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/c9bd2efe5d5c4593925f772659d31c29 2023-04-09 09:56:03,150 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/0d9ea862f2164a0e8d2915e93f3cd501 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/0d9ea862f2164a0e8d2915e93f3cd501 2023-04-09 09:56:03,151 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/7e3567833c804b3db3968083cab07f45 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/7e3567833c804b3db3968083cab07f45 2023-04-09 09:56:03,151 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/a4e36e418b3e4ff98855327e7780a1cc to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/a4e36e418b3e4ff98855327e7780a1cc 2023-04-09 09:56:03,152 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/eee11ccccf93447988aa9d03baedd504 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/eee11ccccf93447988aa9d03baedd504 2023-04-09 09:56:03,153 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/0f037ba4104c4f12a69a41ac4f3f34d5 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/0f037ba4104c4f12a69a41ac4f3f34d5 2023-04-09 09:56:03,154 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/b42216dee8dd451080de07cef69f2083 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/b42216dee8dd451080de07cef69f2083 2023-04-09 09:56:03,155 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/66d87d295c4544f68f2138fbae5281ee to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/66d87d295c4544f68f2138fbae5281ee 2023-04-09 09:56:03,155 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/76aa650003c247beb83aea42f5f03e61 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/76aa650003c247beb83aea42f5f03e61 2023-04-09 09:56:03,156 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/b09bec28124a47038a426d9d757d1e2b to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/b09bec28124a47038a426d9d757d1e2b 2023-04-09 09:56:03,157 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/cddbefe4674e46ef8c9a2cfcb6c2a74e to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/cddbefe4674e46ef8c9a2cfcb6c2a74e 2023-04-09 09:56:03,158 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/3aadb574b7ea48a39e4a1bc72db41022 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/3aadb574b7ea48a39e4a1bc72db41022 2023-04-09 09:56:03,158 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/8483cd68b8b74843ab838e4c89682d13 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/8483cd68b8b74843ab838e4c89682d13 2023-04-09 09:56:03,159 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/d0174daf06b34a5e8aa2d734b18c5fe4 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/d0174daf06b34a5e8aa2d734b18c5fe4 2023-04-09 09:56:03,160 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/20a34cc1f59d4f27856b60a0687c4d13 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/20a34cc1f59d4f27856b60a0687c4d13 2023-04-09 09:56:03,161 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/cdebebce81934e9fa43b2474215ae2ca to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/cdebebce81934e9fa43b2474215ae2ca 2023-04-09 09:56:03,162 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/c3d37b6d3e1046c186779fc4e3c11ff5 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/c3d37b6d3e1046c186779fc4e3c11ff5 2023-04-09 09:56:03,162 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/b1e58e24eef54e16bc7b0a877bf2d790 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/b1e58e24eef54e16bc7b0a877bf2d790 2023-04-09 09:56:03,163 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/e251a28a28744c259915b1dd81bc6adc to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/e251a28a28744c259915b1dd81bc6adc 2023-04-09 09:56:03,164 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/230a11401f0441d29e6c8454dbba3b8c to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/230a11401f0441d29e6c8454dbba3b8c 2023-04-09 09:56:03,165 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/7c9e3ced0699439fb67d36440eaef12c to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/7c9e3ced0699439fb67d36440eaef12c 2023-04-09 09:56:03,165 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/6567c7954953486e8877579f00b1bc34 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/6567c7954953486e8877579f00b1bc34 2023-04-09 09:56:03,166 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/ade3157ea4824c0e9be3f1e14883f4f5 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/ade3157ea4824c0e9be3f1e14883f4f5 2023-04-09 09:56:03,167 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/20e537b474aa44c588ad3d69732713f2 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/20e537b474aa44c588ad3d69732713f2 2023-04-09 09:56:03,168 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/c784667685e04953836405e3c674c294 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/c784667685e04953836405e3c674c294 2023-04-09 09:56:03,169 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/de4eec7029aa4036bd620a99d4fec7ce to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/de4eec7029aa4036bd620a99d4fec7ce 2023-04-09 09:56:03,169 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/32fa6ba826f146f29b7968d35092b4e3 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/32fa6ba826f146f29b7968d35092b4e3 2023-04-09 09:56:03,170 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/9046d34cd4b84d99b54cfa0b62d7b1c1 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/9046d34cd4b84d99b54cfa0b62d7b1c1 2023-04-09 09:56:03,171 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/accb65ed4e9348c2ac901af52ed54730 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/accb65ed4e9348c2ac901af52ed54730 2023-04-09 09:56:03,172 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/41534cc5172b4069b2743301a4081526 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/41534cc5172b4069b2743301a4081526 2023-04-09 09:56:03,172 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/0f8e534b382a44f8b3393433187502dc to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/0f8e534b382a44f8b3393433187502dc 2023-04-09 09:56:03,173 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/48abc2ed111c4b50b033176785aa21cd to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/48abc2ed111c4b50b033176785aa21cd 2023-04-09 09:56:03,174 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/9ded6dad37734164b98a733f2571ff6c to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/9ded6dad37734164b98a733f2571ff6c 2023-04-09 09:56:03,175 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/0ba49e50c10b4e2b9068f37b13629458 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/0ba49e50c10b4e2b9068f37b13629458 2023-04-09 09:56:03,176 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/29bbfaee8bf9469fa5adc9a0bef3f28d to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/29bbfaee8bf9469fa5adc9a0bef3f28d 2023-04-09 09:56:03,177 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/0b9a65f09e9a40a6b94e4aafdb985c56 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/0b9a65f09e9a40a6b94e4aafdb985c56 2023-04-09 09:56:03,178 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/d17a517883fb48208514cb1795e1ab78 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/d17a517883fb48208514cb1795e1ab78 2023-04-09 09:56:03,181 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/bd6937f13a3a42079cf442d73f1b3c20, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/b8ce9346ce7545bba00c6d9d16b094e4, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/47fecacfa69c4fc4a28aeff4cab0d908, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/0aae36b8d7e44545be01d0db6bdcf4df, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/92e11b6c58d04fb9bcf063522a8efedc, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/3d98b5d1614249cd8e4b848e3c4df59d, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/7e9d29ce8d6d4169b93888ade90afb0a, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/d68b211003574a2d909f62b401f8398e, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/b0c76351160c4d7fa431fe929e7edfb4, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/0d414232275a4e2090d78add3f63b5e5, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/f20c9961caf6478e8485884e43e77040, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/ddb2726daf21410899618ccb945e8202, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/fab256d1416f402f855921854ae78b47, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/5f4ec3e0cdb24f5f897a8e578811d26b, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/8e0c0b9dc5214f308c23441c5fc925ee, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/f2aed27732d4414abfb26cbe4c395202, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/65ce40a0eab549dc8d5d1a447ae7ab8c, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/e0186f035c7a4bcd9acbf6acbd512497, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/fea257912ef94f5382a82508be5892c7, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/872e331b74244e05b5d3fc4fce3d7a49, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/f650f6c86ed945a9967d783281d7bbc9, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/9b8c16c693d845568cd38c0e9198c261, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/ccb64889195f4c928785ca9f959f5ff0, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/80bd0acffb5444c4a8e89a475dbc64b1, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/e6f4fdd054444fd7a55b8cd7a41de80b, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/7f7c64de611343a98691eb415352ff32, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/897b5758893940f8ac979ebf516258b6, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/df01f61a1ec1429ea0b551f4665aca74, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/0c5bb1f9f9dc4bec95f6806bf3e56b5d, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/b1d903ec662641b29368389476396ea4, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/2b319e69193542d19e321b63f217cf6d, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/1c5e7f749b314e55a701927c4096b029, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/77bdac29ac504c649732a970de58ce3d, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/7dd5a2bf098f4ee9b9583cec2649dc38, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/9081f967b0f849fa82ab4baa2d3e857c, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/f938136ac3e747d0976e0aab2c3e7236, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/18dce3b200ac48c796c51cc1d7421fc4, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/c4659bdfb8844ef2a3da8aa1d8f86283, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/c53eb5057bad4b9c8cc79653fa633d01, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/3a43eff56fdb4ef58022e3aa86724880, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/5f3e4d7a22c14140a29263d83d34dc06, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/14996b8cd81b49178814a994335d049b, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/cb44afe2b20341f2b9ab7d38f3f12dfb, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/a7c07597f6b04d0594e0f3d90c3dce8d] to archive 2023-04-09 09:56:03,182 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-09 09:56:03,183 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/bd6937f13a3a42079cf442d73f1b3c20 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/bd6937f13a3a42079cf442d73f1b3c20 2023-04-09 09:56:03,184 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/b8ce9346ce7545bba00c6d9d16b094e4 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/b8ce9346ce7545bba00c6d9d16b094e4 2023-04-09 09:56:03,185 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/47fecacfa69c4fc4a28aeff4cab0d908 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/47fecacfa69c4fc4a28aeff4cab0d908 2023-04-09 09:56:03,185 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/0aae36b8d7e44545be01d0db6bdcf4df to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/0aae36b8d7e44545be01d0db6bdcf4df 2023-04-09 09:56:03,186 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/92e11b6c58d04fb9bcf063522a8efedc to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/92e11b6c58d04fb9bcf063522a8efedc 2023-04-09 09:56:03,187 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/3d98b5d1614249cd8e4b848e3c4df59d to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/3d98b5d1614249cd8e4b848e3c4df59d 2023-04-09 09:56:03,188 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/7e9d29ce8d6d4169b93888ade90afb0a to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/7e9d29ce8d6d4169b93888ade90afb0a 2023-04-09 09:56:03,188 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/d68b211003574a2d909f62b401f8398e to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/d68b211003574a2d909f62b401f8398e 2023-04-09 09:56:03,189 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/b0c76351160c4d7fa431fe929e7edfb4 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/b0c76351160c4d7fa431fe929e7edfb4 2023-04-09 09:56:03,190 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/0d414232275a4e2090d78add3f63b5e5 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/0d414232275a4e2090d78add3f63b5e5 2023-04-09 09:56:03,191 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/f20c9961caf6478e8485884e43e77040 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/f20c9961caf6478e8485884e43e77040 2023-04-09 09:56:03,192 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/ddb2726daf21410899618ccb945e8202 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/ddb2726daf21410899618ccb945e8202 2023-04-09 09:56:03,193 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/fab256d1416f402f855921854ae78b47 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/fab256d1416f402f855921854ae78b47 2023-04-09 09:56:03,194 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/5f4ec3e0cdb24f5f897a8e578811d26b to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/5f4ec3e0cdb24f5f897a8e578811d26b 2023-04-09 09:56:03,194 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/8e0c0b9dc5214f308c23441c5fc925ee to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/8e0c0b9dc5214f308c23441c5fc925ee 2023-04-09 09:56:03,195 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/f2aed27732d4414abfb26cbe4c395202 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/f2aed27732d4414abfb26cbe4c395202 2023-04-09 09:56:03,196 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/65ce40a0eab549dc8d5d1a447ae7ab8c to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/65ce40a0eab549dc8d5d1a447ae7ab8c 2023-04-09 09:56:03,197 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/e0186f035c7a4bcd9acbf6acbd512497 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/e0186f035c7a4bcd9acbf6acbd512497 2023-04-09 09:56:03,197 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/fea257912ef94f5382a82508be5892c7 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/fea257912ef94f5382a82508be5892c7 2023-04-09 09:56:03,198 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/872e331b74244e05b5d3fc4fce3d7a49 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/872e331b74244e05b5d3fc4fce3d7a49 2023-04-09 09:56:03,199 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/f650f6c86ed945a9967d783281d7bbc9 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/f650f6c86ed945a9967d783281d7bbc9 2023-04-09 09:56:03,199 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/9b8c16c693d845568cd38c0e9198c261 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/9b8c16c693d845568cd38c0e9198c261 2023-04-09 09:56:03,200 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/ccb64889195f4c928785ca9f959f5ff0 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/ccb64889195f4c928785ca9f959f5ff0 2023-04-09 09:56:03,201 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/80bd0acffb5444c4a8e89a475dbc64b1 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/80bd0acffb5444c4a8e89a475dbc64b1 2023-04-09 09:56:03,202 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/e6f4fdd054444fd7a55b8cd7a41de80b to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/e6f4fdd054444fd7a55b8cd7a41de80b 2023-04-09 09:56:03,202 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/7f7c64de611343a98691eb415352ff32 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/7f7c64de611343a98691eb415352ff32 2023-04-09 09:56:03,203 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/897b5758893940f8ac979ebf516258b6 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/897b5758893940f8ac979ebf516258b6 2023-04-09 09:56:03,204 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/df01f61a1ec1429ea0b551f4665aca74 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/df01f61a1ec1429ea0b551f4665aca74 2023-04-09 09:56:03,205 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/0c5bb1f9f9dc4bec95f6806bf3e56b5d to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/0c5bb1f9f9dc4bec95f6806bf3e56b5d 2023-04-09 09:56:03,206 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/b1d903ec662641b29368389476396ea4 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/b1d903ec662641b29368389476396ea4 2023-04-09 09:56:03,206 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/2b319e69193542d19e321b63f217cf6d to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/2b319e69193542d19e321b63f217cf6d 2023-04-09 09:56:03,207 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/1c5e7f749b314e55a701927c4096b029 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/1c5e7f749b314e55a701927c4096b029 2023-04-09 09:56:03,208 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/77bdac29ac504c649732a970de58ce3d to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/77bdac29ac504c649732a970de58ce3d 2023-04-09 09:56:03,209 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/7dd5a2bf098f4ee9b9583cec2649dc38 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/7dd5a2bf098f4ee9b9583cec2649dc38 2023-04-09 09:56:03,209 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/9081f967b0f849fa82ab4baa2d3e857c to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/9081f967b0f849fa82ab4baa2d3e857c 2023-04-09 09:56:03,210 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/f938136ac3e747d0976e0aab2c3e7236 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/f938136ac3e747d0976e0aab2c3e7236 2023-04-09 09:56:03,211 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/18dce3b200ac48c796c51cc1d7421fc4 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/18dce3b200ac48c796c51cc1d7421fc4 2023-04-09 09:56:03,212 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/c4659bdfb8844ef2a3da8aa1d8f86283 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/c4659bdfb8844ef2a3da8aa1d8f86283 2023-04-09 09:56:03,213 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/c53eb5057bad4b9c8cc79653fa633d01 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/c53eb5057bad4b9c8cc79653fa633d01 2023-04-09 09:56:03,214 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/3a43eff56fdb4ef58022e3aa86724880 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/3a43eff56fdb4ef58022e3aa86724880 2023-04-09 09:56:03,214 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/5f3e4d7a22c14140a29263d83d34dc06 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/5f3e4d7a22c14140a29263d83d34dc06 2023-04-09 09:56:03,215 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/14996b8cd81b49178814a994335d049b to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/14996b8cd81b49178814a994335d049b 2023-04-09 09:56:03,216 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/cb44afe2b20341f2b9ab7d38f3f12dfb to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/cb44afe2b20341f2b9ab7d38f3f12dfb 2023-04-09 09:56:03,217 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/a7c07597f6b04d0594e0f3d90c3dce8d to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/a7c07597f6b04d0594e0f3d90c3dce8d 2023-04-09 09:56:03,220 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/1dcb7efa637c4a20a749bb782e2d646f, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/1da5ee03214744829ed767d86420c4f3, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/36e278ca57bf495ba3b86734b3401779, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/4b274876648540ff91277c8cf90982c5, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/dee0848ce6324834b89e15700dd0703c, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/d1b03bebd97c4b11876568049a0cfed6, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/da297746c817402fbc14e6e570d0e6ca, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/549df63dab2140d1b35d9d2a7fa96b0c, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/d8db39d6e8e742f4bb39934ff3fd1d87, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/0b65f9bd30b549578d159dbf6a47ca3e, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/1f67a71e84ee44f9b2c3ef1a6326a073, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/be8fa7f8c284429fad8e59f920b854f5, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/9c35607a98664a6bb5ad65b69d70ea8f, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/d07a76ec0b72488bb7aa3d89eeba0ab8, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/e8af8ded416944b0852dd00cdc61689b, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/647c66655efd4bd9a7e8d1be179d7293, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/b0ee312e0dad46909d8690187b0246b4, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/a485f890737f489baf78df2141d90890, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/ddac8b65087f438c87b0ac46bf99c85c, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/22d7eb899f5f4d64a43b805192d09e8f, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/ddae70465267481b912c19d13ebf9477, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/841654606d954311bc940b34ad580abc, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/466013a3c4f840f6bda2bdb2094a39f3, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/3660b8cde2914331988e32c2c52e3bac, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/f8aae433e960426d9a7de0833e0d70d1, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/0f92c62541bc46e29c0d15560397f27c, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/dee7aecf53d94bfe8036780b006f6dcc, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/f45909998c0d4fc7b46e9ee0e347901d, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/87947492ff71469eb3f2521b02f79b2f, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/624d59e44fe64ab6a8cfbfa1799b629d, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/dcb64d88866344aaba909576b211ac7f, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/a0e4a221fc854a0d946a1923019b46df, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/bb4e2f5bbc3849b9a3e94c5ca7cff084, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/e25e526b7a0e4350a756e977f35c6ceb, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/51a816e2764b4fba8ef0a6119b45e0a6, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/a8b018d403ce4ddb9d9c3bbce17b6552, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/5419ea9f45764d09b29444986d77f596, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/56942368a0e648be9beef5ef67df0c07, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/155bbfdf6f3643caaedd0d4579ed5663, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/6d5a94f4c2224455939b9f0740fdca44, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/3ec990e42a224608975e57d2b143754a, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/91c73064316c463c93922d7b9599746f, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/68fedc8964554ff3954f890d31704664, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/f103666abaef4742966d23362b62092f] to archive 2023-04-09 09:56:03,220 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-09 09:56:03,221 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/1dcb7efa637c4a20a749bb782e2d646f to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/1dcb7efa637c4a20a749bb782e2d646f 2023-04-09 09:56:03,222 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/1da5ee03214744829ed767d86420c4f3 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/1da5ee03214744829ed767d86420c4f3 2023-04-09 09:56:03,223 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/36e278ca57bf495ba3b86734b3401779 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/36e278ca57bf495ba3b86734b3401779 2023-04-09 09:56:03,223 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/4b274876648540ff91277c8cf90982c5 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/4b274876648540ff91277c8cf90982c5 2023-04-09 09:56:03,224 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/dee0848ce6324834b89e15700dd0703c to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/dee0848ce6324834b89e15700dd0703c 2023-04-09 09:56:03,225 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/d1b03bebd97c4b11876568049a0cfed6 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/d1b03bebd97c4b11876568049a0cfed6 2023-04-09 09:56:03,226 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/da297746c817402fbc14e6e570d0e6ca to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/da297746c817402fbc14e6e570d0e6ca 2023-04-09 09:56:03,226 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/549df63dab2140d1b35d9d2a7fa96b0c to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/549df63dab2140d1b35d9d2a7fa96b0c 2023-04-09 09:56:03,227 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/d8db39d6e8e742f4bb39934ff3fd1d87 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/d8db39d6e8e742f4bb39934ff3fd1d87 2023-04-09 09:56:03,228 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/0b65f9bd30b549578d159dbf6a47ca3e to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/0b65f9bd30b549578d159dbf6a47ca3e 2023-04-09 09:56:03,229 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/1f67a71e84ee44f9b2c3ef1a6326a073 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/1f67a71e84ee44f9b2c3ef1a6326a073 2023-04-09 09:56:03,229 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/be8fa7f8c284429fad8e59f920b854f5 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/be8fa7f8c284429fad8e59f920b854f5 2023-04-09 09:56:03,230 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/9c35607a98664a6bb5ad65b69d70ea8f to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/9c35607a98664a6bb5ad65b69d70ea8f 2023-04-09 09:56:03,231 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/d07a76ec0b72488bb7aa3d89eeba0ab8 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/d07a76ec0b72488bb7aa3d89eeba0ab8 2023-04-09 09:56:03,232 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/e8af8ded416944b0852dd00cdc61689b to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/e8af8ded416944b0852dd00cdc61689b 2023-04-09 09:56:03,233 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/647c66655efd4bd9a7e8d1be179d7293 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/647c66655efd4bd9a7e8d1be179d7293 2023-04-09 09:56:03,234 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/b0ee312e0dad46909d8690187b0246b4 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/b0ee312e0dad46909d8690187b0246b4 2023-04-09 09:56:03,235 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/a485f890737f489baf78df2141d90890 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/a485f890737f489baf78df2141d90890 2023-04-09 09:56:03,236 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/ddac8b65087f438c87b0ac46bf99c85c to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/ddac8b65087f438c87b0ac46bf99c85c 2023-04-09 09:56:03,237 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/22d7eb899f5f4d64a43b805192d09e8f to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/22d7eb899f5f4d64a43b805192d09e8f 2023-04-09 09:56:03,237 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/ddae70465267481b912c19d13ebf9477 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/ddae70465267481b912c19d13ebf9477 2023-04-09 09:56:03,238 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/841654606d954311bc940b34ad580abc to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/841654606d954311bc940b34ad580abc 2023-04-09 09:56:03,239 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/466013a3c4f840f6bda2bdb2094a39f3 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/466013a3c4f840f6bda2bdb2094a39f3 2023-04-09 09:56:03,240 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/3660b8cde2914331988e32c2c52e3bac to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/3660b8cde2914331988e32c2c52e3bac 2023-04-09 09:56:03,241 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/f8aae433e960426d9a7de0833e0d70d1 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/f8aae433e960426d9a7de0833e0d70d1 2023-04-09 09:56:03,242 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/0f92c62541bc46e29c0d15560397f27c to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/0f92c62541bc46e29c0d15560397f27c 2023-04-09 09:56:03,242 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/dee7aecf53d94bfe8036780b006f6dcc to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/dee7aecf53d94bfe8036780b006f6dcc 2023-04-09 09:56:03,243 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/f45909998c0d4fc7b46e9ee0e347901d to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/f45909998c0d4fc7b46e9ee0e347901d 2023-04-09 09:56:03,244 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/87947492ff71469eb3f2521b02f79b2f to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/87947492ff71469eb3f2521b02f79b2f 2023-04-09 09:56:03,244 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/624d59e44fe64ab6a8cfbfa1799b629d to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/624d59e44fe64ab6a8cfbfa1799b629d 2023-04-09 09:56:03,245 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/dcb64d88866344aaba909576b211ac7f to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/dcb64d88866344aaba909576b211ac7f 2023-04-09 09:56:03,246 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/a0e4a221fc854a0d946a1923019b46df to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/a0e4a221fc854a0d946a1923019b46df 2023-04-09 09:56:03,246 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/bb4e2f5bbc3849b9a3e94c5ca7cff084 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/bb4e2f5bbc3849b9a3e94c5ca7cff084 2023-04-09 09:56:03,247 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/e25e526b7a0e4350a756e977f35c6ceb to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/e25e526b7a0e4350a756e977f35c6ceb 2023-04-09 09:56:03,248 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/51a816e2764b4fba8ef0a6119b45e0a6 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/51a816e2764b4fba8ef0a6119b45e0a6 2023-04-09 09:56:03,248 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/a8b018d403ce4ddb9d9c3bbce17b6552 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/a8b018d403ce4ddb9d9c3bbce17b6552 2023-04-09 09:56:03,249 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/5419ea9f45764d09b29444986d77f596 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/5419ea9f45764d09b29444986d77f596 2023-04-09 09:56:03,250 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/56942368a0e648be9beef5ef67df0c07 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/56942368a0e648be9beef5ef67df0c07 2023-04-09 09:56:03,250 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/155bbfdf6f3643caaedd0d4579ed5663 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/155bbfdf6f3643caaedd0d4579ed5663 2023-04-09 09:56:03,251 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/6d5a94f4c2224455939b9f0740fdca44 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/6d5a94f4c2224455939b9f0740fdca44 2023-04-09 09:56:03,252 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/3ec990e42a224608975e57d2b143754a to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/3ec990e42a224608975e57d2b143754a 2023-04-09 09:56:03,252 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/91c73064316c463c93922d7b9599746f to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/91c73064316c463c93922d7b9599746f 2023-04-09 09:56:03,253 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/68fedc8964554ff3954f890d31704664 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/68fedc8964554ff3954f890d31704664 2023-04-09 09:56:03,254 DEBUG [StoreCloser-TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/f103666abaef4742966d23362b62092f to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/f103666abaef4742966d23362b62092f 2023-04-09 09:56:03,257 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/recovered.edits/930.seqid, newMaxSeqId=930, maxSeqId=1 2023-04-09 09:56:03,258 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00. 2023-04-09 09:56:03,258 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 2764d7f8155e1e5b0d641eebe063fa00: 2023-04-09 09:56:03,259 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 2764d7f8155e1e5b0d641eebe063fa00 2023-04-09 09:56:03,260 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=2764d7f8155e1e5b0d641eebe063fa00, regionState=CLOSED 2023-04-09 09:56:03,260 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681034163260"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681034163260"}]},"ts":"1681034163260"} 2023-04-09 09:56:03,262 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=52, resume processing ppid=51 2023-04-09 09:56:03,262 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=51, state=SUCCESS; CloseRegionProcedure 2764d7f8155e1e5b0d641eebe063fa00, server=jenkins-hbase4.apache.org,42467,1681034019544 in 1.5310 sec 2023-04-09 09:56:03,263 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=50 2023-04-09 09:56:03,263 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=50, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=2764d7f8155e1e5b0d641eebe063fa00, UNASSIGN in 1.5350 sec 2023-04-09 09:56:03,263 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681034163263"}]},"ts":"1681034163263"} 2023-04-09 09:56:03,264 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-09 09:56:03,265 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-09 09:56:03,266 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=50, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5440 sec 2023-04-09 09:56:03,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-09 09:56:03,829 INFO [Listener at localhost/44563] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 50 completed 2023-04-09 09:56:03,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-09 09:56:03,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure2.ProcedureExecutor(1029): Stored pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-09 09:56:03,832 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-09 09:56:03,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-09 09:56:03,832 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=53, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-09 09:56:03,837 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00 2023-04-09 09:56:03,839 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A, FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B, FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C, FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/recovered.edits] 2023-04-09 09:56:03,842 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/4f365d8c365f49ddb0ff5292287f41a9 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/4f365d8c365f49ddb0ff5292287f41a9 2023-04-09 09:56:03,843 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/dd4b9c4065e34d65adb673f0404e4bb8 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/A/dd4b9c4065e34d65adb673f0404e4bb8 2023-04-09 09:56:03,845 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/73f9ea996e5f4f3eb66adde2073599fa to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/73f9ea996e5f4f3eb66adde2073599fa 2023-04-09 09:56:03,846 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/c6171ba2a70549a3ab47363a40916b24 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/B/c6171ba2a70549a3ab47363a40916b24 2023-04-09 09:56:03,847 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/0731e11aad7b414dae4338943446af08 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/0731e11aad7b414dae4338943446af08 2023-04-09 09:56:03,848 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/f62a9f9514684bc2aadec973fa854c42 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/C/f62a9f9514684bc2aadec973fa854c42 2023-04-09 09:56:03,850 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/recovered.edits/930.seqid to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00/recovered.edits/930.seqid 2023-04-09 09:56:03,851 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/2764d7f8155e1e5b0d641eebe063fa00 2023-04-09 09:56:03,851 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-09 09:56:03,853 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=53, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-09 09:56:03,854 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-09 09:56:03,856 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-09 09:56:03,857 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=53, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-09 09:56:03,857 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-09 09:56:03,857 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681034163857"}]},"ts":"9223372036854775807"} 2023-04-09 09:56:03,858 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-09 09:56:03,858 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 2764d7f8155e1e5b0d641eebe063fa00, NAME => 'TestAcidGuarantees,,1681034137020.2764d7f8155e1e5b0d641eebe063fa00.', STARTKEY => '', ENDKEY => ''}] 2023-04-09 09:56:03,858 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-09 09:56:03,858 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681034163858"}]},"ts":"9223372036854775807"} 2023-04-09 09:56:03,859 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-09 09:56:03,861 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=53, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-09 09:56:03,861 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=53, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 31 msec 2023-04-09 09:56:03,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-09 09:56:03,933 INFO [Listener at localhost/44563] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 53 completed 2023-04-09 09:56:03,945 INFO [Listener at localhost/44563] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=229 (was 230), OpenFileDescriptor=560 (was 546) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=374 (was 386), ProcessCount=169 (was 169), AvailableMemoryMB=1525 (was 1437) - AvailableMemoryMB LEAK? - 2023-04-09 09:56:03,957 INFO [Listener at localhost/44563] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=229, OpenFileDescriptor=560, MaxFileDescriptor=60000, SystemLoadAverage=374, ProcessCount=169, AvailableMemoryMB=1525 2023-04-09 09:56:03,960 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-09 09:56:03,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-09 09:56:03,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure2.ProcedureExecutor(1029): Stored pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-09 09:56:03,962 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-09 09:56:03,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 54 2023-04-09 09:56:03,963 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-09 09:56:03,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-09 09:56:03,964 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95 2023-04-09 09:56:03,964 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95 empty. 2023-04-09 09:56:03,965 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95 2023-04-09 09:56:03,965 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-09 09:56:04,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-09 09:56:04,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-09 09:56:04,372 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-09 09:56:04,373 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => c8127616cf5355dd7f1db99764192b95, NAME => 'TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp 2023-04-09 09:56:04,388 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-09 09:56:04,389 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing c8127616cf5355dd7f1db99764192b95, disabling compactions & flushes 2023-04-09 09:56:04,389 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:04,389 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:04,389 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. after waiting 0 ms 2023-04-09 09:56:04,389 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:04,389 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:04,389 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for c8127616cf5355dd7f1db99764192b95: 2023-04-09 09:56:04,391 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-09 09:56:04,392 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681034164392"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681034164392"}]},"ts":"1681034164392"} 2023-04-09 09:56:04,393 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-09 09:56:04,394 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-09 09:56:04,394 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681034164394"}]},"ts":"1681034164394"} 2023-04-09 09:56:04,395 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-09 09:56:04,398 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c8127616cf5355dd7f1db99764192b95, ASSIGN}] 2023-04-09 09:56:04,400 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c8127616cf5355dd7f1db99764192b95, ASSIGN 2023-04-09 09:56:04,401 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=c8127616cf5355dd7f1db99764192b95, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,42467,1681034019544; forceNewPlan=false, retain=false 2023-04-09 09:56:04,552 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=c8127616cf5355dd7f1db99764192b95, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:04,552 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681034164552"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681034164552"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681034164552"}]},"ts":"1681034164552"} 2023-04-09 09:56:04,553 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=55, state=RUNNABLE; OpenRegionProcedure c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544}] 2023-04-09 09:56:04,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-09 09:56:04,707 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:04,708 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => c8127616cf5355dd7f1db99764192b95, NAME => 'TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.', STARTKEY => '', ENDKEY => ''} 2023-04-09 09:56:04,708 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees c8127616cf5355dd7f1db99764192b95 2023-04-09 09:56:04,708 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-09 09:56:04,708 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for c8127616cf5355dd7f1db99764192b95 2023-04-09 09:56:04,708 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for c8127616cf5355dd7f1db99764192b95 2023-04-09 09:56:04,709 INFO [StoreOpener-c8127616cf5355dd7f1db99764192b95-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region c8127616cf5355dd7f1db99764192b95 2023-04-09 09:56:04,710 DEBUG [StoreOpener-c8127616cf5355dd7f1db99764192b95-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A 2023-04-09 09:56:04,710 DEBUG [StoreOpener-c8127616cf5355dd7f1db99764192b95-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A 2023-04-09 09:56:04,711 INFO [StoreOpener-c8127616cf5355dd7f1db99764192b95-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-09 09:56:04,711 INFO [StoreOpener-c8127616cf5355dd7f1db99764192b95-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c8127616cf5355dd7f1db99764192b95 columnFamilyName A 2023-04-09 09:56:04,711 INFO [StoreOpener-c8127616cf5355dd7f1db99764192b95-1] regionserver.HStore(310): Store=c8127616cf5355dd7f1db99764192b95/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-09 09:56:04,712 INFO [StoreOpener-c8127616cf5355dd7f1db99764192b95-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region c8127616cf5355dd7f1db99764192b95 2023-04-09 09:56:04,713 DEBUG [StoreOpener-c8127616cf5355dd7f1db99764192b95-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B 2023-04-09 09:56:04,713 DEBUG [StoreOpener-c8127616cf5355dd7f1db99764192b95-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B 2023-04-09 09:56:04,713 INFO [StoreOpener-c8127616cf5355dd7f1db99764192b95-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-09 09:56:04,713 INFO [StoreOpener-c8127616cf5355dd7f1db99764192b95-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c8127616cf5355dd7f1db99764192b95 columnFamilyName B 2023-04-09 09:56:04,714 INFO [StoreOpener-c8127616cf5355dd7f1db99764192b95-1] regionserver.HStore(310): Store=c8127616cf5355dd7f1db99764192b95/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-09 09:56:04,714 INFO [StoreOpener-c8127616cf5355dd7f1db99764192b95-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region c8127616cf5355dd7f1db99764192b95 2023-04-09 09:56:04,715 DEBUG [StoreOpener-c8127616cf5355dd7f1db99764192b95-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C 2023-04-09 09:56:04,715 DEBUG [StoreOpener-c8127616cf5355dd7f1db99764192b95-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C 2023-04-09 09:56:04,715 INFO [StoreOpener-c8127616cf5355dd7f1db99764192b95-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-09 09:56:04,715 INFO [StoreOpener-c8127616cf5355dd7f1db99764192b95-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c8127616cf5355dd7f1db99764192b95 columnFamilyName C 2023-04-09 09:56:04,716 INFO [StoreOpener-c8127616cf5355dd7f1db99764192b95-1] regionserver.HStore(310): Store=c8127616cf5355dd7f1db99764192b95/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-09 09:56:04,716 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:04,716 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95 2023-04-09 09:56:04,717 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95 2023-04-09 09:56:04,718 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-09 09:56:04,719 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for c8127616cf5355dd7f1db99764192b95 2023-04-09 09:56:04,721 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-09 09:56:04,721 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened c8127616cf5355dd7f1db99764192b95; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60141795, jitterRate=-0.103817418217659}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-09 09:56:04,721 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for c8127616cf5355dd7f1db99764192b95: 2023-04-09 09:56:04,722 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95., pid=56, masterSystemTime=1681034164705 2023-04-09 09:56:04,723 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:04,723 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:04,723 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=c8127616cf5355dd7f1db99764192b95, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:04,724 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681034164723"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681034164723"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681034164723"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681034164723"}]},"ts":"1681034164723"} 2023-04-09 09:56:04,725 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=55 2023-04-09 09:56:04,726 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=55, state=SUCCESS; OpenRegionProcedure c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 in 171 msec 2023-04-09 09:56:04,727 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=54 2023-04-09 09:56:04,727 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=54, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=c8127616cf5355dd7f1db99764192b95, ASSIGN in 328 msec 2023-04-09 09:56:04,727 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-09 09:56:04,728 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681034164727"}]},"ts":"1681034164727"} 2023-04-09 09:56:04,728 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-09 09:56:04,730 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-09 09:56:04,731 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=54, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 770 msec 2023-04-09 09:56:05,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-09 09:56:05,067 INFO [Listener at localhost/44563] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 54 completed 2023-04-09 09:56:05,067 DEBUG [Listener at localhost/44563] zookeeper.ReadOnlyZKClient(139): Connect 0x470742aa to 127.0.0.1:57816 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-09 09:56:05,072 DEBUG [Listener at localhost/44563] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3ced4ba, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-09 09:56:05,073 DEBUG [Listener at localhost/44563] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-09 09:56:05,074 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60420, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-09 09:56:05,075 DEBUG [Listener at localhost/44563] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-09 09:56:05,077 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48318, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-09 09:56:05,079 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-09 09:56:05,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-09 09:56:05,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure2.ProcedureExecutor(1029): Stored pid=57, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-09 09:56:05,090 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-09 09:56:05,091 DEBUG [PEWorker-4] util.FSTableDescriptors(519): Deleted hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-09 09:56:05,091 INFO [PEWorker-4] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-09 09:56:05,093 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=57, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-09 09:56:05,095 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c8127616cf5355dd7f1db99764192b95, REOPEN/MOVE}] 2023-04-09 09:56:05,095 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c8127616cf5355dd7f1db99764192b95, REOPEN/MOVE 2023-04-09 09:56:05,096 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=c8127616cf5355dd7f1db99764192b95, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:05,096 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681034165096"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681034165096"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681034165096"}]},"ts":"1681034165096"} 2023-04-09 09:56:05,097 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=59, state=RUNNABLE; CloseRegionProcedure c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544}] 2023-04-09 09:56:05,248 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close c8127616cf5355dd7f1db99764192b95 2023-04-09 09:56:05,249 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing c8127616cf5355dd7f1db99764192b95, disabling compactions & flushes 2023-04-09 09:56:05,249 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:05,249 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:05,249 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. after waiting 0 ms 2023-04-09 09:56:05,249 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:05,252 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-09 09:56:05,253 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:05,253 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for c8127616cf5355dd7f1db99764192b95: 2023-04-09 09:56:05,253 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: c8127616cf5355dd7f1db99764192b95 to self. 2023-04-09 09:56:05,254 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed c8127616cf5355dd7f1db99764192b95 2023-04-09 09:56:05,254 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=c8127616cf5355dd7f1db99764192b95, regionState=CLOSED 2023-04-09 09:56:05,254 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681034165254"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681034165254"}]},"ts":"1681034165254"} 2023-04-09 09:56:05,256 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=59 2023-04-09 09:56:05,256 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=59, state=SUCCESS; CloseRegionProcedure c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 in 158 msec 2023-04-09 09:56:05,257 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=c8127616cf5355dd7f1db99764192b95, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,42467,1681034019544; forceNewPlan=false, retain=true 2023-04-09 09:56:05,407 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=c8127616cf5355dd7f1db99764192b95, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:05,407 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681034165407"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681034165407"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681034165407"}]},"ts":"1681034165407"} 2023-04-09 09:56:05,409 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=59, state=RUNNABLE; OpenRegionProcedure c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544}] 2023-04-09 09:56:05,563 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:05,563 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => c8127616cf5355dd7f1db99764192b95, NAME => 'TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.', STARTKEY => '', ENDKEY => ''} 2023-04-09 09:56:05,563 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees c8127616cf5355dd7f1db99764192b95 2023-04-09 09:56:05,563 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-09 09:56:05,563 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for c8127616cf5355dd7f1db99764192b95 2023-04-09 09:56:05,563 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for c8127616cf5355dd7f1db99764192b95 2023-04-09 09:56:05,564 INFO [StoreOpener-c8127616cf5355dd7f1db99764192b95-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region c8127616cf5355dd7f1db99764192b95 2023-04-09 09:56:05,565 DEBUG [StoreOpener-c8127616cf5355dd7f1db99764192b95-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A 2023-04-09 09:56:05,565 DEBUG [StoreOpener-c8127616cf5355dd7f1db99764192b95-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A 2023-04-09 09:56:05,565 INFO [StoreOpener-c8127616cf5355dd7f1db99764192b95-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-09 09:56:05,565 INFO [StoreOpener-c8127616cf5355dd7f1db99764192b95-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c8127616cf5355dd7f1db99764192b95 columnFamilyName A 2023-04-09 09:56:05,566 INFO [StoreOpener-c8127616cf5355dd7f1db99764192b95-1] regionserver.HStore(310): Store=c8127616cf5355dd7f1db99764192b95/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-09 09:56:05,567 INFO [StoreOpener-c8127616cf5355dd7f1db99764192b95-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region c8127616cf5355dd7f1db99764192b95 2023-04-09 09:56:05,567 DEBUG [StoreOpener-c8127616cf5355dd7f1db99764192b95-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B 2023-04-09 09:56:05,567 DEBUG [StoreOpener-c8127616cf5355dd7f1db99764192b95-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B 2023-04-09 09:56:05,567 INFO [StoreOpener-c8127616cf5355dd7f1db99764192b95-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-09 09:56:05,568 INFO [StoreOpener-c8127616cf5355dd7f1db99764192b95-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c8127616cf5355dd7f1db99764192b95 columnFamilyName B 2023-04-09 09:56:05,568 INFO [StoreOpener-c8127616cf5355dd7f1db99764192b95-1] regionserver.HStore(310): Store=c8127616cf5355dd7f1db99764192b95/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-09 09:56:05,568 INFO [StoreOpener-c8127616cf5355dd7f1db99764192b95-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region c8127616cf5355dd7f1db99764192b95 2023-04-09 09:56:05,569 DEBUG [StoreOpener-c8127616cf5355dd7f1db99764192b95-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C 2023-04-09 09:56:05,569 DEBUG [StoreOpener-c8127616cf5355dd7f1db99764192b95-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C 2023-04-09 09:56:05,569 INFO [StoreOpener-c8127616cf5355dd7f1db99764192b95-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-09 09:56:05,569 INFO [StoreOpener-c8127616cf5355dd7f1db99764192b95-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c8127616cf5355dd7f1db99764192b95 columnFamilyName C 2023-04-09 09:56:05,569 INFO [StoreOpener-c8127616cf5355dd7f1db99764192b95-1] regionserver.HStore(310): Store=c8127616cf5355dd7f1db99764192b95/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-09 09:56:05,569 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:05,570 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95 2023-04-09 09:56:05,571 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95 2023-04-09 09:56:05,573 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-09 09:56:05,573 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for c8127616cf5355dd7f1db99764192b95 2023-04-09 09:56:05,574 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened c8127616cf5355dd7f1db99764192b95; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71707151, jitterRate=0.06851981580257416}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-09 09:56:05,574 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for c8127616cf5355dd7f1db99764192b95: 2023-04-09 09:56:05,574 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95., pid=61, masterSystemTime=1681034165560 2023-04-09 09:56:05,576 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:05,576 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:05,576 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=c8127616cf5355dd7f1db99764192b95, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:05,576 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681034165576"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681034165576"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681034165576"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681034165576"}]},"ts":"1681034165576"} 2023-04-09 09:56:05,578 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=59 2023-04-09 09:56:05,578 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=59, state=SUCCESS; OpenRegionProcedure c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 in 168 msec 2023-04-09 09:56:05,579 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=58 2023-04-09 09:56:05,579 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=58, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=c8127616cf5355dd7f1db99764192b95, REOPEN/MOVE in 483 msec 2023-04-09 09:56:05,580 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=57 2023-04-09 09:56:05,580 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=57, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 487 msec 2023-04-09 09:56:05,581 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=57, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 502 msec 2023-04-09 09:56:05,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=57 2023-04-09 09:56:05,583 DEBUG [Listener at localhost/44563] zookeeper.ReadOnlyZKClient(139): Connect 0x1e3503b4 to 127.0.0.1:57816 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-09 09:56:05,587 DEBUG [Listener at localhost/44563] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@63da10ca, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-09 09:56:05,587 DEBUG [Listener at localhost/44563] zookeeper.ReadOnlyZKClient(139): Connect 0x6f06dd50 to 127.0.0.1:57816 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-09 09:56:05,592 DEBUG [Listener at localhost/44563] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@59185ba7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-09 09:56:05,592 DEBUG [Listener at localhost/44563] zookeeper.ReadOnlyZKClient(139): Connect 0x5f9e8654 to 127.0.0.1:57816 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-09 09:56:05,597 DEBUG [Listener at localhost/44563] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@62b3caa, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-09 09:56:05,597 DEBUG [Listener at localhost/44563] zookeeper.ReadOnlyZKClient(139): Connect 0x4db22237 to 127.0.0.1:57816 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-09 09:56:05,600 DEBUG [Listener at localhost/44563] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@66b5b1f3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-09 09:56:05,600 DEBUG [Listener at localhost/44563] zookeeper.ReadOnlyZKClient(139): Connect 0x08b57aae to 127.0.0.1:57816 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-09 09:56:05,604 DEBUG [Listener at localhost/44563] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@544a4215, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-09 09:56:05,604 DEBUG [Listener at localhost/44563] zookeeper.ReadOnlyZKClient(139): Connect 0x25baee3e to 127.0.0.1:57816 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-09 09:56:05,608 DEBUG [Listener at localhost/44563] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@23e65a76, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-09 09:56:05,609 DEBUG [Listener at localhost/44563] zookeeper.ReadOnlyZKClient(139): Connect 0x2c53bf96 to 127.0.0.1:57816 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-09 09:56:05,614 DEBUG [Listener at localhost/44563] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5e6a6f3c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-09 09:56:05,614 DEBUG [Listener at localhost/44563] zookeeper.ReadOnlyZKClient(139): Connect 0x3b59bdcf to 127.0.0.1:57816 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-09 09:56:05,618 DEBUG [Listener at localhost/44563] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@a470a3d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-09 09:56:05,618 DEBUG [Listener at localhost/44563] zookeeper.ReadOnlyZKClient(139): Connect 0x3d765193 to 127.0.0.1:57816 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-09 09:56:05,622 DEBUG [Listener at localhost/44563] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4bb72437, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-09 09:56:05,622 DEBUG [Listener at localhost/44563] zookeeper.ReadOnlyZKClient(139): Connect 0x32bf16cd to 127.0.0.1:57816 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-09 09:56:05,627 DEBUG [Listener at localhost/44563] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3dca23c0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-09 09:56:05,631 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:05,636 DEBUG [hconnection-0x6b4608c9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-09 09:56:05,639 DEBUG [hconnection-0x31d0f30e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-09 09:56:05,639 DEBUG [hconnection-0x6ee678f1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-09 09:56:05,639 DEBUG [hconnection-0x53f7fd54-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-09 09:56:05,640 DEBUG [hconnection-0x2167f2b9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-09 09:56:05,640 DEBUG [hconnection-0x37a82022-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-09 09:56:05,640 DEBUG [hconnection-0x47bd0f09-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-09 09:56:05,640 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60460, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-09 09:56:05,644 DEBUG [hconnection-0x6d794f3c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-09 09:56:05,648 DEBUG [hconnection-0x6b9368bf-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-09 09:56:05,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:05,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:05,644 DEBUG [hconnection-0x7d5d04e6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-09 09:56:05,649 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:05,649 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:05,649 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:05,649 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:05,649 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60498, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-09 09:56:05,649 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60436, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-09 09:56:05,649 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60476, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-09 09:56:05,649 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60448, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-09 09:56:05,651 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60512, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-09 09:56:05,651 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60482, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-09 09:56:05,655 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60506, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-09 09:56:05,654 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:05,658 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:05,658 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60516, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-09 09:56:05,660 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:05,660 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:05,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:05,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:05,661 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:05,661 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:05,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:05,661 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:05,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:05,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:05,661 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:05,662 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:05,662 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:05,663 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:05,665 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:05,665 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:05,666 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60508, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-09 09:56:05,666 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:05,667 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:05,667 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:05,667 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:05,668 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c8127616cf5355dd7f1db99764192b95 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-04-09 09:56:05,669 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=A 2023-04-09 09:56:05,669 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:56:05,669 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=B 2023-04-09 09:56:05,669 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:56:05,669 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=C 2023-04-09 09:56:05,669 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:56:05,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on c8127616cf5355dd7f1db99764192b95 2023-04-09 09:56:05,671 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:05,683 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034165677, max timestamp=1681034165681Num uniques 859; 2023-04-09 09:56:05,683 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1681034165677, max timestamp=1681034165681Num uniques 859; 2023-04-09 09:56:05,683 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:56:05,683 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7005, suffix heap size=117064,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=153 2023-04-09 09:56:05,684 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681034165677, max timestamp=1681034165680Num uniques 859; 2023-04-09 09:56:05,684 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-09 09:56:05,689 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040907ee4fd54a484662a5eaabd2d055c2f0 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040907ee4fd54a484662a5eaabd2d055c2f0 2023-04-09 09:56:05,695 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=11, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/A/057ab254f6ce439692bb82f8f79f86b2 2023-04-09 09:56:05,697 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7684, totalHeapSize=17712, min timestamp=1681034165685, max timestamp=1681034165696Num uniques 1018; 2023-04-09 09:56:05,697 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46614, new segment data size=7684, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-09 09:56:05,698 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681034165685, max timestamp=1681034165696Num uniques 1012; 2023-04-09 09:56:05,698 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46348, new segment data size=7688, suffix heap size=133216,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=168 2023-04-09 09:56:05,698 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681034165685, max timestamp=1681034165696Num uniques 1009; 2023-04-09 09:56:05,698 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=7688, suffix heap size=132904,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=168 2023-04-09 09:56:05,708 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/129c073ee3e84e3cbc8d2a4fafcdc7bd 2023-04-09 09:56:05,715 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681034165709, max timestamp=1681034165713Num uniques 1027; 2023-04-09 09:56:05,715 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-09 09:56:05,715 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681034165709, max timestamp=1681034165713Num uniques 1027; 2023-04-09 09:56:05,716 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-09 09:56:05,715 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681034165709, max timestamp=1681034165713Num uniques 1027; 2023-04-09 09:56:05,716 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-09 09:56:05,721 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/C/74b917e760f448caa7aad394f8a80885 2023-04-09 09:56:05,724 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/A/057ab254f6ce439692bb82f8f79f86b2 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/057ab254f6ce439692bb82f8f79f86b2 2023-04-09 09:56:05,728 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/057ab254f6ce439692bb82f8f79f86b2, entries=150, sequenceid=11, filesize=25.1 K 2023-04-09 09:56:05,728 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/129c073ee3e84e3cbc8d2a4fafcdc7bd as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/129c073ee3e84e3cbc8d2a4fafcdc7bd 2023-04-09 09:56:05,730 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681034165722, max timestamp=1681034165728Num uniques 1036; 2023-04-09 09:56:05,730 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681034165722, max timestamp=1681034165728Num uniques 1036; 2023-04-09 09:56:05,730 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-09 09:56:05,730 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-09 09:56:05,730 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681034165722, max timestamp=1681034165728Num uniques 1036; 2023-04-09 09:56:05,730 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-09 09:56:05,733 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/129c073ee3e84e3cbc8d2a4fafcdc7bd, entries=150, sequenceid=11, filesize=11.6 K 2023-04-09 09:56:05,734 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/C/74b917e760f448caa7aad394f8a80885 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/74b917e760f448caa7aad394f8a80885 2023-04-09 09:56:05,742 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/74b917e760f448caa7aad394f8a80885, entries=150, sequenceid=11, filesize=11.6 K 2023-04-09 09:56:05,743 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=114.05 KB/116790 for c8127616cf5355dd7f1db99764192b95 in 75ms, sequenceid=11, compaction requested=false 2023-04-09 09:56:05,743 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c8127616cf5355dd7f1db99764192b95: 2023-04-09 09:56:05,743 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:05,743 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:05,743 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:05,743 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:05,743 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:05,743 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:05,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on c8127616cf5355dd7f1db99764192b95 2023-04-09 09:56:05,744 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c8127616cf5355dd7f1db99764192b95 3/3 column families, dataSize=120.76 KB heapSize=337.92 KB 2023-04-09 09:56:05,744 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:05,744 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:05,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:05,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:05,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:05,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:05,745 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=A 2023-04-09 09:56:05,745 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:05,746 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:56:05,746 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=B 2023-04-09 09:56:05,746 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:05,746 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:56:05,746 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=C 2023-04-09 09:56:05,746 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:56:05,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:05,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:05,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:05,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:05,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:05,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@293791e3[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:05,748 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:05,748 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:05,750 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:05,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:05,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:05,750 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:05,750 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:05,750 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:05,750 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:05,750 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:05,754 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:05,754 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:05,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60498 deadline: 1681034225749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:05,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60506 deadline: 1681034225750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:05,755 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:05,755 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:05,755 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:05,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60448 deadline: 1681034225753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:05,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:05,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:05,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:05,755 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:05,756 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:05,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:05,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:05,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:05,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:05,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:05,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:05,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:05,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:05,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:05,759 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:05,759 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:05,759 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:05,759 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:05,759 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:05,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60436 deadline: 1681034225755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:05,759 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:05,759 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:05,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60460 deadline: 1681034225755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:05,766 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:05,766 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:05,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:05,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:05,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:05,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:05,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:05,766 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:05,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:05,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:05,767 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:05,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:05,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:05,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:05,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:05,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:05,768 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:05,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:05,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:05,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:05,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:05,770 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:05,770 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:05,770 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:05,770 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:05,770 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:56:05,770 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:05,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:05,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:05,770 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:05,771 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:05,771 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:05,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:05,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:05,771 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:05,771 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:05,771 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:05,771 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:05,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:05,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:05,857 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:05,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60498 deadline: 1681034225856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:05,857 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:05,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60506 deadline: 1681034225856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:05,857 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:05,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60448 deadline: 1681034225856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:05,861 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:05,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60436 deadline: 1681034225860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:05,862 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:05,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60460 deadline: 1681034225860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:05,871 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:05,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:05,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:05,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:05,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:05,874 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:05,875 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:05,875 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:05,875 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:05,875 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:05,875 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:05,876 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:05,877 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:05,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:05,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:05,877 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:05,877 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:05,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:05,877 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:05,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:05,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:05,878 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:05,878 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:05,878 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:05,878 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:05,878 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:05,878 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:05,878 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:05,878 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:05,878 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:05,879 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:05,879 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:05,879 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:05,879 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:05,879 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:05,879 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:05,879 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:05,879 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:05,881 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:05,881 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:05,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:05,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:05,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:05,881 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:05,881 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:05,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:05,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:05,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:05,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:05,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:05,883 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:05,883 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1653173c[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:05,883 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:05,883 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:05,884 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:05,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:05,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:05,885 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:05,885 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:05,885 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:05,885 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:05,885 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:05,886 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:05,886 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:05,886 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:05,886 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:05,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:05,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:05,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:05,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:05,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:05,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:05,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:05,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:05,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:05,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:05,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:05,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:05,889 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:05,889 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:05,889 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:05,889 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:05,889 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:05,890 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:05,890 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:05,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:05,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:05,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:05,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:05,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:05,890 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:05,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:05,891 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:05,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:05,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:05,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:05,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:05,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:05,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:05,892 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:05,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:05,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:05,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:05,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:05,894 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:05,894 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:05,894 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:05,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:05,894 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:05,894 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:05,894 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:05,894 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:05,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:05,895 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:05,895 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:05,895 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:05,895 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:05,895 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:05,895 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:05,895 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:05,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:05,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:05,895 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:56:05,995 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:05,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:05,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:05,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:05,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:05,998 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:05,998 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:05,998 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:05,998 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:05,999 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:05,999 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:06,000 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:06,000 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:06,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:06,001 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:06,001 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:06,005 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:06,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:06,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:06,006 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:06,006 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:06,006 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:06,006 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:06,006 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:06,006 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:06,006 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:06,006 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:06,006 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:06,006 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:06,007 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:06,007 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:06,007 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:06,007 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:06,007 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,007 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:06,007 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:06,008 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,008 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:06,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:06,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:06,009 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:06,009 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:06,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:06,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:06,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:06,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:06,010 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:06,010 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d43a88f[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:06,010 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:06,010 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:06,011 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:06,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:06,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:06,011 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:06,011 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:06,011 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,012 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:06,011 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:06,013 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,013 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:06,013 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:06,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:06,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:06,013 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:06,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:06,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:06,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:06,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:06,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:06,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:06,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:06,015 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:06,016 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:06,016 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:06,016 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:06,016 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:06,018 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:06,018 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:06,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:06,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:06,018 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:06,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:06,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:06,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:06,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:06,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:06,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:06,019 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:06,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:06,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:06,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:06,020 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:06,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:06,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,023 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,023 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:06,023 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:06,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:06,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:06,023 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,023 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:06,023 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:06,023 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:06,023 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:06,023 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:06,023 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:06,024 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:06,024 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:06,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:06,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:06,024 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:56:06,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:06,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:06,060 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:06,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60448 deadline: 1681034226058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,060 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:06,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60506 deadline: 1681034226058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,060 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:06,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60498 deadline: 1681034226059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,064 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:06,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60436 deadline: 1681034226063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,064 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:06,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60460 deadline: 1681034226063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,124 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:06,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:06,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:06,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:06,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:06,127 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:06,127 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:06,127 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:06,127 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:06,128 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:06,128 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:06,129 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,129 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:06,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:06,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:06,129 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:06,129 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:06,130 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:06,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:06,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:06,130 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:06,130 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:06,130 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:06,130 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:06,130 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:06,130 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:06,131 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:06,131 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:06,131 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:06,131 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:06,131 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:06,131 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:06,131 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:06,131 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,131 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:06,131 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:06,133 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:06,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:06,133 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:06,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:06,133 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:06,133 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:06,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:06,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:06,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:06,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:06,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6335ece6[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:06,135 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:06,135 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:06,141 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:06,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:06,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:06,141 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:06,142 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:06,142 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:06,142 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,142 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:06,143 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:06,143 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,143 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:06,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:06,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:06,143 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:06,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:06,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:06,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:06,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:06,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:06,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:06,145 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:06,145 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:06,145 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:06,145 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:06,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:06,145 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:06,147 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:06,147 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:06,147 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:06,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:06,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:06,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:06,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:06,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:06,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:06,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:06,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:06,148 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:06,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:06,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:06,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:06,149 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:06,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:06,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,155 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:06,155 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:06,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:06,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:06,155 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:06,155 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:06,155 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:06,155 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:06,155 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:06,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:06,155 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:56:06,156 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:06,155 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,156 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:06,156 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:06,156 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:06,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:06,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:06,177 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040974d4e1d97a1e4587bc463c62e9a69f32 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040974d4e1d97a1e4587bc463c62e9a69f32 2023-04-09 09:56:06,256 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:06,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:06,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:06,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:06,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:06,260 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:06,260 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:06,260 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:06,260 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:06,260 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:06,260 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:06,261 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,261 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:06,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:06,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:06,261 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,261 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:06,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:06,262 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:06,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:06,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:06,263 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:06,263 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:06,263 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:06,263 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:06,263 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:06,263 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:06,264 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:06,264 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:06,264 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:06,264 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:06,264 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:06,264 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:06,264 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:06,264 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:06,265 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,265 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:06,265 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:06,266 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,266 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:06,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:06,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:06,266 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:06,266 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:06,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:06,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:06,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:06,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:06,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:06,267 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:06,268 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:06,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7618d1ea[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:06,269 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:06,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:06,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:06,269 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:06,269 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:06,269 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:06,269 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,270 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:06,271 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,271 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:06,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:06,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:06,272 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:06,272 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:06,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:06,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:06,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:06,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:06,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:06,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:06,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:06,274 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:06,275 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:06,275 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:06,275 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:06,275 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:06,276 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:06,276 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:06,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:06,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:06,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:06,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:06,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:06,276 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:06,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:06,276 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:06,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:06,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:06,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:06,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:06,278 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:06,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:06,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:06,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,280 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:06,280 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:06,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:06,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:06,280 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,280 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:06,280 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:06,280 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:06,280 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:06,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:06,280 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:06,280 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:06,280 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:06,281 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:06,281 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:06,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:06,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:06,281 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:56:06,364 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:06,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60448 deadline: 1681034226362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,364 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:06,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60506 deadline: 1681034226362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,364 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:06,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60498 deadline: 1681034226362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,366 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:06,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60436 deadline: 1681034226365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,368 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:06,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60460 deadline: 1681034226367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,381 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:06,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:06,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:06,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:06,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:06,384 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:06,384 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:06,384 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:06,384 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:06,384 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:06,384 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:06,385 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,385 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:06,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:06,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:06,386 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,386 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:06,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:06,386 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:06,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:06,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:06,387 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:06,387 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:06,387 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:06,387 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:06,387 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:06,388 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:06,388 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:06,388 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:06,388 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:06,388 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:06,388 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:06,388 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:06,389 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:06,389 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:06,389 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,389 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:06,389 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:06,391 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,391 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:06,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:06,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:06,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:06,392 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:06,392 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:06,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:06,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:06,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:06,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:06,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78fcd2e2[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:06,394 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:06,394 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:06,395 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:06,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:06,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:06,395 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:06,395 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:06,395 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:06,395 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,395 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:06,397 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,397 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:06,397 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:06,397 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:06,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:06,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:06,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:06,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:06,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:06,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:06,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:06,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:06,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:06,400 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:06,400 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:06,400 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:06,400 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:06,400 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:06,402 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:06,402 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:06,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:06,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:06,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:06,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:06,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:06,402 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:06,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:06,403 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:06,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:06,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:06,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:06,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:06,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:06,404 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:06,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:06,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,407 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,407 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:06,407 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:06,407 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:06,407 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:06,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:06,407 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:06,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:06,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:06,408 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:56:06,407 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:06,407 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:06,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:06,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:06,407 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:06,408 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:06,408 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:06,408 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:06,508 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:06,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:06,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:06,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:06,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:06,512 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:06,512 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:06,512 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:06,512 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:06,512 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:06,512 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:06,513 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:06,513 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:06,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:06,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:06,514 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,514 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:06,514 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:06,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:06,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:06,515 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:06,515 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:06,515 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:06,515 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:06,515 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:06,515 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:06,515 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:06,515 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:06,515 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:06,515 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:06,515 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:06,516 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:06,516 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:06,516 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:06,516 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,516 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:06,516 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:06,517 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,517 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:06,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:06,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:06,518 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:06,518 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:06,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:06,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:06,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:06,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:06,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:06,520 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@484b1aa2[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:06,520 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:06,520 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:06,521 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:06,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:06,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:06,521 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:06,521 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:06,521 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:06,521 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,521 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:06,523 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:06,523 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,523 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:06,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:06,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:06,523 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:06,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:06,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:06,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:06,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:06,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:06,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:06,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:06,525 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:06,525 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:06,525 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:06,525 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:06,525 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:06,526 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:06,526 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:06,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:06,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:06,526 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:06,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:06,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:06,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:06,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:06,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:06,527 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:06,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:06,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:06,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:06,528 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:06,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:06,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:06,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,530 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,530 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:06,530 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:06,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:06,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:06,530 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:06,530 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:06,530 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:06,530 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:06,530 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:06,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:06,530 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:06,530 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:56:06,530 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:06,531 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:06,531 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:06,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:06,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:06,582 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=97, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/A/8e0c3f1efbb849f89322be02af6cb4b3 2023-04-09 09:56:06,631 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:06,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:06,633 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:06,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:06,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:06,635 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:06,635 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:06,635 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:06,635 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:06,635 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:06,635 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:06,640 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,640 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:06,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:06,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:06,640 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,640 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:06,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:06,641 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:06,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:06,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:06,641 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:06,642 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:06,642 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:06,642 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:06,642 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:06,642 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:06,642 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:06,642 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:06,642 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:06,642 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:06,643 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:06,643 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:06,643 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:06,643 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:06,643 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,643 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:06,643 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:06,644 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,644 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:06,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:06,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:06,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:06,645 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:06,645 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:06,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:06,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:06,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:06,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:06,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d59d91[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:06,647 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:06,647 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:06,649 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:06,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:06,649 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,649 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:06,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:06,649 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:06,649 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:06,649 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:06,651 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,651 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:06,651 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:06,651 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:06,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:06,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:06,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:06,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:06,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:06,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:06,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:06,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:06,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:06,654 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:06,654 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:06,654 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:06,654 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:06,654 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:06,656 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:06,656 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:06,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:06,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:06,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:06,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:06,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:06,660 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:06,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:06,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:06,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:06,661 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:06,662 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:06,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:06,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:06,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:06,664 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:06,664 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:06,664 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,664 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:06,664 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:06,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:06,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:06,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:06,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:06,664 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:06,664 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:06,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:06,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:06,664 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,664 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:06,665 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:06,664 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:56:06,665 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:06,665 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:06,764 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:06,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:06,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:06,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:06,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:06,772 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:06,772 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:06,772 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:06,772 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:06,773 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:06,773 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:06,774 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:06,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:06,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:06,774 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:06,775 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,775 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:06,775 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:06,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:06,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:06,775 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:06,776 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:06,776 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:06,776 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:06,776 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:06,776 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:06,776 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:06,776 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:06,776 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:06,777 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:06,777 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:06,777 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:06,777 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:06,777 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:06,777 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,777 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:06,777 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:06,779 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:06,779 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:06,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:06,779 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:06,779 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:06,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:06,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:06,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:06,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:06,781 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:06,781 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34729b60[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:06,781 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:06,781 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:06,784 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:06,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:06,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:06,784 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:06,784 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:06,784 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:06,784 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,784 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:06,785 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,785 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:06,786 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:06,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:06,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:06,786 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:06,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:06,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:06,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:06,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:06,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:06,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:06,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:06,788 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:06,788 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:06,788 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:06,788 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:06,789 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:06,791 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:06,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:06,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:06,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:06,791 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:06,791 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:06,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:06,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:06,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:06,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:06,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:06,792 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:06,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:06,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:06,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:06,793 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:06,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:06,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,796 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:06,796 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:06,797 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:06,797 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,797 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:06,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:06,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:06,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:06,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:06,797 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:06,797 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,797 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:06,797 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:06,797 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:06,797 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:06,797 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:06,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:06,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:06,797 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:56:06,866 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:06,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60506 deadline: 1681034226865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,869 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:06,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60498 deadline: 1681034226866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,870 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:06,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60448 deadline: 1681034226868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,871 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:06,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60436 deadline: 1681034226870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,875 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:06,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60460 deadline: 1681034226873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,897 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:06,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:06,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:06,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:06,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:06,900 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:06,900 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:06,900 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:06,900 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:06,901 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:06,901 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:06,903 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,903 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:06,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:06,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:06,903 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,903 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:06,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:06,903 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:06,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:06,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:06,904 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:06,904 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:06,904 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:06,904 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:06,904 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:06,904 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:06,904 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:06,904 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:06,904 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:06,904 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:06,905 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:06,905 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:06,905 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:06,905 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:06,905 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,905 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:06,905 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:06,906 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,906 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:06,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:06,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:06,906 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:06,906 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:06,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:06,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:06,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:06,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:06,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:06,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a0724ce[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:06,908 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:06,908 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:06,909 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:06,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:06,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:06,909 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:06,909 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,909 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:06,909 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:06,909 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:06,910 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:06,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:06,910 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:06,910 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:06,911 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:06,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:06,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:06,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:06,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:06,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:06,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:06,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:06,913 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:06,913 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:06,913 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:06,913 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:06,913 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:06,915 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:06,915 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:06,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:06,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:06,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:06,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:06,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:06,915 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:06,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:06,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:06,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:06,915 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:06,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:06,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:06,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:06,916 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:06,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:06,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,918 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,918 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:06,918 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:06,918 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:06,918 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:06,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:06,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:06,918 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:06,919 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:06,918 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:56:06,918 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:06,918 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:06,919 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:06,919 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:06,919 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:06,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:06,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:06,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:06,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:07,002 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=97 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/051d433cf1324af5a626244822009bde 2023-04-09 09:56:07,011 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=97 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/C/c8b220063d05497ca6ca47ea08124297 2023-04-09 09:56:07,014 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/A/8e0c3f1efbb849f89322be02af6cb4b3 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/8e0c3f1efbb849f89322be02af6cb4b3 2023-04-09 09:56:07,017 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/8e0c3f1efbb849f89322be02af6cb4b3, entries=150, sequenceid=97, filesize=25.1 K 2023-04-09 09:56:07,018 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/051d433cf1324af5a626244822009bde as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/051d433cf1324af5a626244822009bde 2023-04-09 09:56:07,019 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:07,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:07,021 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/051d433cf1324af5a626244822009bde, entries=150, sequenceid=97, filesize=11.6 K 2023-04-09 09:56:07,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:07,021 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/C/c8b220063d05497ca6ca47ea08124297 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/c8b220063d05497ca6ca47ea08124297 2023-04-09 09:56:07,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:07,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:07,022 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:07,022 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:07,022 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:07,022 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:07,023 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:07,023 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:07,024 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:07,024 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:07,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:07,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:07,025 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:07,025 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:07,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:07,025 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:07,025 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/c8b220063d05497ca6ca47ea08124297, entries=150, sequenceid=97, filesize=11.6 K 2023-04-09 09:56:07,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:07,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:07,025 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:07,026 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:07,026 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:07,026 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:07,026 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:07,026 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~357.80 KB/366384, currentSize=53.67 KB/54960 for c8127616cf5355dd7f1db99764192b95 in 1282ms, sequenceid=97, compaction requested=false 2023-04-09 09:56:07,026 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:07,026 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c8127616cf5355dd7f1db99764192b95: 2023-04-09 09:56:07,026 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:07,026 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:07,026 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:07,026 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:07,026 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c8127616cf5355dd7f1db99764192b95 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-09 09:56:07,026 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=A 2023-04-09 09:56:07,027 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:56:07,027 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=B 2023-04-09 09:56:07,027 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:56:07,027 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=C 2023-04-09 09:56:07,027 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:56:07,448 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304094ad77b4095534703a2cd0cc4b91749a5 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304094ad77b4095534703a2cd0cc4b91749a5 2023-04-09 09:56:07,854 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=108, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/A/f7a366128bf842cdb23569c24cd5d86c 2023-04-09 09:56:07,863 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=108 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/a763996c14b5440789cadce1a146905f 2023-04-09 09:56:07,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on c8127616cf5355dd7f1db99764192b95 2023-04-09 09:56:07,873 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:07,891 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681034167881, max timestamp=1681034167890Num uniques 859; 2023-04-09 09:56:07,891 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681034167881, max timestamp=1681034167890Num uniques 859; 2023-04-09 09:56:07,891 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-09 09:56:07,891 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-09 09:56:07,891 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681034167881, max timestamp=1681034167890Num uniques 859; 2023-04-09 09:56:07,891 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-09 09:56:07,901 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=108 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/C/d88ac10f0b5a4045a38d2e5441f5c8f0 2023-04-09 09:56:07,904 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:07,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60448 deadline: 1681034227900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:07,905 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/A/f7a366128bf842cdb23569c24cd5d86c as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/f7a366128bf842cdb23569c24cd5d86c 2023-04-09 09:56:07,905 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:07,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60498 deadline: 1681034227901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:07,906 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:07,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60460 deadline: 1681034227902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:07,906 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:07,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60506 deadline: 1681034227903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:07,907 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:07,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60436 deadline: 1681034227904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:07,908 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/f7a366128bf842cdb23569c24cd5d86c, entries=150, sequenceid=108, filesize=25.1 K 2023-04-09 09:56:07,909 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/a763996c14b5440789cadce1a146905f as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/a763996c14b5440789cadce1a146905f 2023-04-09 09:56:07,912 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/a763996c14b5440789cadce1a146905f, entries=150, sequenceid=108, filesize=11.6 K 2023-04-09 09:56:07,913 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/C/d88ac10f0b5a4045a38d2e5441f5c8f0 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/d88ac10f0b5a4045a38d2e5441f5c8f0 2023-04-09 09:56:07,916 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/d88ac10f0b5a4045a38d2e5441f5c8f0, entries=150, sequenceid=108, filesize=11.6 K 2023-04-09 09:56:07,916 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for c8127616cf5355dd7f1db99764192b95 in 890ms, sequenceid=108, compaction requested=true 2023-04-09 09:56:07,916 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c8127616cf5355dd7f1db99764192b95: 2023-04-09 09:56:07,916 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:07,916 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:07,916 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:07,917 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:07,917 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:07,917 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:07,925 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:07,926 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:07,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:07,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:07,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:07,926 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:07,926 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:07,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:07,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:07,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:07,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:07,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:07,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:07,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a89a526[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:07,927 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:07,927 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:07,928 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:07,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:07,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:07,929 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:07,929 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:07,929 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:07,929 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:07,929 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:07,931 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:07,931 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:07,931 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:07,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:07,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:07,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:07,931 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:07,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:07,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:07,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:07,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:07,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:07,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:07,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:07,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:07,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:07,934 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:07,934 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:07,934 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:07,934 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:07,934 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:07,935 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:07,935 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:07,935 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:07,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:07,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:07,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:07,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:07,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:07,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:07,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:07,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:07,936 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:07,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:07,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:07,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:07,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:07,937 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:07,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:07,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:07,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:07,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:07,939 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:07,939 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:07,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:07,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:07,939 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:07,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:07,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:07,939 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:07,939 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:07,939 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:07,939 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:07,940 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:07,940 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:07,939 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:07,940 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:07,940 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:07,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:07,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:08,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on c8127616cf5355dd7f1db99764192b95 2023-04-09 09:56:08,007 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c8127616cf5355dd7f1db99764192b95 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-04-09 09:56:08,007 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=A 2023-04-09 09:56:08,007 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:56:08,007 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=B 2023-04-09 09:56:08,007 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:56:08,007 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=C 2023-04-09 09:56:08,007 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:56:08,017 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:08,017 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:08,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60436 deadline: 1681034228013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60506 deadline: 1681034228012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,018 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:08,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60498 deadline: 1681034228016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,021 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:08,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60460 deadline: 1681034228017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,024 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:08,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60448 deadline: 1681034228022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,040 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:08,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:08,045 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:08,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:08,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:08,046 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:08,046 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:08,046 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:08,046 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:08,047 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:08,047 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:08,048 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:08,048 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:08,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:08,048 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,048 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:08,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:08,055 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:08,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:08,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:08,056 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:08,059 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:08,059 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:08,059 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:08,059 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:08,059 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:08,059 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:08,059 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:08,059 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:08,059 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:08,059 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:08,059 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:08,059 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:08,060 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:08,060 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,060 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:08,060 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:08,066 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,066 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:08,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:08,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:08,066 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:08,067 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:08,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:08,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:08,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:08,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:08,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:08,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12989ef3[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:08,070 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:08,070 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:08,071 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:08,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:08,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:08,072 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:08,072 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:08,072 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:08,072 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,072 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:08,073 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:08,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:08,073 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:08,073 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:08,073 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:08,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:08,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:08,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:08,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:08,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:08,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:08,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:08,078 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:08,078 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:08,078 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:08,078 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:08,078 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:08,080 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:08,080 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:08,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:08,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:08,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:08,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:08,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:08,082 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:08,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:08,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:08,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:08,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:08,083 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:08,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:08,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:08,085 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:08,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:08,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,088 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:08,088 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:08,088 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,088 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:08,088 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:08,088 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,088 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:08,088 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:08,088 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:08,088 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:56:08,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:08,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:08,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:08,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:08,088 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:08,089 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:08,089 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:08,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:08,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:08,120 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:08,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60498 deadline: 1681034228119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,122 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:08,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60436 deadline: 1681034228121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,123 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:08,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60460 deadline: 1681034228122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,124 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:08,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60506 deadline: 1681034228122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,126 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:08,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60448 deadline: 1681034228125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,189 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:08,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:08,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:08,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:08,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:08,192 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:08,192 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:08,192 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:08,192 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:08,192 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:08,192 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:08,193 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:08,193 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:08,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:08,194 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,194 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:08,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:08,194 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:08,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:08,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:08,194 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:08,195 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:08,195 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:08,195 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:08,195 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:08,195 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:08,195 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:08,195 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:08,195 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:08,195 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:08,195 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:08,195 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:08,195 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:08,195 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:08,195 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,196 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:08,196 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:08,197 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,197 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:08,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:08,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:08,197 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:08,197 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:08,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:08,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:08,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:08,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:08,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:08,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3211be5b[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:08,199 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:08,199 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:08,199 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:08,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:08,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:08,200 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:08,200 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:08,200 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:08,200 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,200 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:08,202 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,202 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:08,203 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:08,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:08,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:08,203 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:08,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:08,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:08,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:08,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:08,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:08,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:08,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:08,205 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:08,205 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:08,205 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:08,205 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:08,205 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:08,206 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:08,206 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:08,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:08,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:08,206 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:08,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:08,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:08,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:08,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:08,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:08,207 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:08,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:08,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:08,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:08,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:08,208 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:08,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:08,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,210 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:08,210 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:08,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:08,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:08,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:08,210 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:08,211 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:08,211 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,211 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:08,211 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:08,211 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:08,210 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:08,211 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:08,210 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:08,211 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:08,211 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:56:08,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:08,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:08,311 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:08,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:08,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:08,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:08,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:08,316 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:08,316 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:08,316 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:08,316 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:08,316 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:08,316 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:08,318 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,318 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:08,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:08,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:08,318 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,318 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:08,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:08,318 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:08,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:08,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:08,319 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:08,319 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:08,319 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:08,319 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:08,319 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:08,319 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:08,319 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:08,319 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:08,319 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:08,320 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:08,320 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:08,320 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:08,320 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:08,320 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,320 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:08,320 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:08,323 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:08,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60498 deadline: 1681034228321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,324 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,324 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:08,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:08,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:08,325 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:08,325 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:08,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:08,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:08,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:08,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,326 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:08,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60460 deadline: 1681034228325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:08,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:08,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21a3dd83[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:08,327 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:08,327 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:08,328 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:08,328 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:08,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60448 deadline: 1681034228327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:08,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:08,329 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:08,329 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:08,329 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:08,329 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:08,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60436 deadline: 1681034228328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,329 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,329 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:08,332 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:08,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60506 deadline: 1681034228329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,336 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:08,336 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:08,336 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:08,336 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:08,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:08,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:08,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:08,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:08,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:08,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:08,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:08,339 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:08,339 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:08,339 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:08,339 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:08,339 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:08,339 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:08,342 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:08,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:08,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:08,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:08,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:08,342 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:08,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:08,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:08,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,344 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,344 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:08,344 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:08,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:08,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:08,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:08,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:08,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:08,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:08,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:08,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,347 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:08,347 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:08,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:08,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:08,347 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,347 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:08,347 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:08,347 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,347 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:08,347 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:08,347 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:08,347 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:08,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:08,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:08,347 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:08,347 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:08,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:08,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:08,422 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304093f95f5fee39c46d7bb36141b60e59959 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304093f95f5fee39c46d7bb36141b60e59959 2023-04-09 09:56:08,427 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=143, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/A/f3f6fd2b45e647a5922bd4ea4a561cb4 2023-04-09 09:56:08,439 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=143 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/09b07cc2f6b548d4bb5ebfb8e94a74e1 2023-04-09 09:56:08,453 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:08,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:08,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:08,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:08,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:08,456 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:08,456 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:08,456 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:08,456 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:08,456 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:08,456 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:08,457 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:08,457 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:08,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:08,457 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,457 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:08,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:08,458 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:08,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:08,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:08,458 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:08,458 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:08,458 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:08,458 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:08,458 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:08,458 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:08,460 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:08,460 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:08,460 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:08,460 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:08,460 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:08,460 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:08,460 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:08,460 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:08,460 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,460 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:08,460 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:08,461 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,461 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:08,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:08,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:08,462 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:08,462 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:08,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:08,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:08,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:08,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:08,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:08,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56f4ec04[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:08,463 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:08,463 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:08,465 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:08,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:08,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:08,465 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:08,465 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:08,465 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:08,469 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:08,469 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:08,469 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:08,469 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,469 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:08,469 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:08,469 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:08,469 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:08,469 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:08,469 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:08,469 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:08,471 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:08,471 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:08,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:08,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:08,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:08,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:08,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:08,471 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,472 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,475 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:08,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:08,475 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:08,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:08,475 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:08,476 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:08,476 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,476 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:08,476 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:08,476 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,476 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:08,476 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:08,476 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:08,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:08,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:08,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:08,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:08,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:08,476 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:08,476 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:08,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:08,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:08,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:08,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:08,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:08,478 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:56:08,577 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:08,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:08,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:08,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:08,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:08,580 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:08,580 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:08,580 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:08,580 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:08,581 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:08,581 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:08,583 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:08,583 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:08,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:08,583 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,583 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:08,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:08,583 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:08,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:08,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:08,584 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:08,584 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:08,584 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:08,584 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:08,584 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:08,584 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:08,584 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:08,584 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:08,584 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:08,584 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:08,585 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:08,585 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:08,585 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:08,585 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:08,585 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,585 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:08,585 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:08,586 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:08,586 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:08,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:08,586 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:08,586 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:08,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:08,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:08,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:08,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:08,587 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:08,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@cd9f2f[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:08,588 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:08,588 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:08,588 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:08,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:08,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:08,589 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:08,589 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:08,589 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:08,589 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,589 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:08,590 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,590 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:08,590 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:08,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:08,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:08,590 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:08,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:08,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:08,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:08,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:08,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:08,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:08,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:08,592 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:08,592 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:08,592 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:08,592 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:08,592 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:08,595 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:08,595 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:08,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:08,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:08,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:08,595 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:08,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:08,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:08,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:08,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:08,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:08,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:08,595 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:08,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:08,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:08,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,596 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:08,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:08,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,598 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,598 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:08,598 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:08,598 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,598 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:08,598 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:08,599 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:08,599 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:08,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:08,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:08,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:08,599 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:08,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:08,599 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:08,599 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:08,599 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:08,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:08,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:08,628 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:08,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60498 deadline: 1681034228626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,630 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:08,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60460 deadline: 1681034228627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,633 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:08,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60436 deadline: 1681034228630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,634 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:08,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60448 deadline: 1681034228630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,640 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:08,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60506 deadline: 1681034228637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,699 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:08,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:08,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:08,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:08,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:08,702 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:08,702 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:08,702 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:08,702 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:08,703 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:08,703 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:08,704 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:08,704 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:08,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:08,704 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:08,704 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:08,705 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:08,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:08,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:08,705 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:08,705 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:08,705 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:08,705 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:08,705 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:08,705 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:08,705 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:08,705 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:08,705 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:08,706 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:08,706 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:08,706 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:08,706 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:08,706 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:08,706 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,706 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:08,706 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:08,707 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,707 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:08,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:08,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:08,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:08,707 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:08,707 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:08,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:08,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:08,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:08,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:08,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f110a0b[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:08,709 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:08,709 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:08,710 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:08,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:08,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:08,710 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,711 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:08,710 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:08,711 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:08,711 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:08,712 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,712 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:08,712 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:08,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:08,712 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:08,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:08,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:08,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:08,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:08,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:08,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:08,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:08,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:08,714 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:08,714 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:08,714 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:08,714 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:08,714 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:08,719 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:08,719 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:08,719 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:08,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:08,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:08,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:08,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:08,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:08,719 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:08,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:08,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:08,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:08,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:08,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:08,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:08,721 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:08,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:08,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,724 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,724 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:08,724 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:08,724 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:08,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:08,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:08,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:08,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:08,724 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:08,724 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:08,724 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,724 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:08,724 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:08,724 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:08,724 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:08,724 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:08,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:08,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:08,799 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-09 09:56:08,824 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:08,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:08,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:08,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:08,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:08,837 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:08,838 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:08,838 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:08,838 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:08,838 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:08,838 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:08,839 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:08,839 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:08,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:08,840 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:08,840 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:08,840 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:08,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:08,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:08,840 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:08,848 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:08,848 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:08,848 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:08,848 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:08,848 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:08,860 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:08,860 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:08,860 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:08,860 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:08,860 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=143 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/C/9e49d1dbf7ca4ffc87d8cbfaa3cf445e 2023-04-09 09:56:08,860 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:08,860 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:08,860 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:08,860 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:08,860 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,860 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:08,860 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:08,862 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:08,862 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:08,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:08,863 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:08,863 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:08,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:08,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:08,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:08,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,864 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/A/f3f6fd2b45e647a5922bd4ea4a561cb4 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/f3f6fd2b45e647a5922bd4ea4a561cb4 2023-04-09 09:56:08,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:08,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:08,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6197af[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:08,865 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:08,865 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:08,866 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:08,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:08,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:08,866 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:08,866 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:08,866 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:08,866 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,866 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:08,868 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/f3f6fd2b45e647a5922bd4ea4a561cb4, entries=200, sequenceid=143, filesize=32.1 K 2023-04-09 09:56:08,868 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,868 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:08,868 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:08,868 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:08,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:08,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:08,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:08,869 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/09b07cc2f6b548d4bb5ebfb8e94a74e1 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/09b07cc2f6b548d4bb5ebfb8e94a74e1 2023-04-09 09:56:08,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:08,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:08,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:08,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:08,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:08,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:08,872 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:08,872 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:08,872 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:08,872 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:08,872 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:08,872 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/09b07cc2f6b548d4bb5ebfb8e94a74e1, entries=150, sequenceid=143, filesize=11.8 K 2023-04-09 09:56:08,873 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/C/9e49d1dbf7ca4ffc87d8cbfaa3cf445e as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/9e49d1dbf7ca4ffc87d8cbfaa3cf445e 2023-04-09 09:56:08,874 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:08,874 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:08,874 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:08,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:08,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:08,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:08,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:08,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:08,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:08,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:08,875 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:08,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:08,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:08,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:08,877 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/9e49d1dbf7ca4ffc87d8cbfaa3cf445e, entries=150, sequenceid=143, filesize=11.8 K 2023-04-09 09:56:08,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:08,877 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=53.67 KB/54960 for c8127616cf5355dd7f1db99764192b95 in 870ms, sequenceid=143, compaction requested=true 2023-04-09 09:56:08,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,878 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c8127616cf5355dd7f1db99764192b95: 2023-04-09 09:56:08,878 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,878 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:56:08,878 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:56:08,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:08,878 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-09 09:56:08,878 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-09 09:56:08,878 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-09 09:56:08,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:08,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,879 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 110124 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-09 09:56:08,879 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47694 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-09 09:56:08,879 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): c8127616cf5355dd7f1db99764192b95/A is initiating minor compaction (all files) 2023-04-09 09:56:08,880 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1912): c8127616cf5355dd7f1db99764192b95/B is initiating minor compaction (all files) 2023-04-09 09:56:08,880 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c8127616cf5355dd7f1db99764192b95/A in TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:08,880 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c8127616cf5355dd7f1db99764192b95/B in TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:08,880 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/057ab254f6ce439692bb82f8f79f86b2, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/8e0c3f1efbb849f89322be02af6cb4b3, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/f7a366128bf842cdb23569c24cd5d86c, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/f3f6fd2b45e647a5922bd4ea4a561cb4] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp, totalSize=107.5 K 2023-04-09 09:56:08,880 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/129c073ee3e84e3cbc8d2a4fafcdc7bd, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/051d433cf1324af5a626244822009bde, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/a763996c14b5440789cadce1a146905f, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/09b07cc2f6b548d4bb5ebfb8e94a74e1] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp, totalSize=46.6 K 2023-04-09 09:56:08,880 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 057ab254f6ce439692bb82f8f79f86b2, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1681034165658 2023-04-09 09:56:08,880 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 129c073ee3e84e3cbc8d2a4fafcdc7bd, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1681034165658 2023-04-09 09:56:08,880 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 8e0c3f1efbb849f89322be02af6cb4b3, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1681034165739 2023-04-09 09:56:08,881 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:08,881 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,881 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:08,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:08,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:08,881 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:08,881 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:08,881 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,881 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:08,881 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:08,881 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting f7a366128bf842cdb23569c24cd5d86c, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1681034165749 2023-04-09 09:56:08,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:08,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:08,881 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:08,881 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:08,881 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:08,882 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting f3f6fd2b45e647a5922bd4ea4a561cb4, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1681034167895 2023-04-09 09:56:08,881 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 051d433cf1324af5a626244822009bde, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1681034165739 2023-04-09 09:56:08,882 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:08,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:08,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:08,882 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting a763996c14b5440789cadce1a146905f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1681034165749 2023-04-09 09:56:08,882 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 09b07cc2f6b548d4bb5ebfb8e94a74e1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1681034167895 2023-04-09 09:56:08,893 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] throttle.PressureAwareThroughputController(145): c8127616cf5355dd7f1db99764192b95#B#compaction#568 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:56:08,897 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): c8127616cf5355dd7f1db99764192b95#A#compaction#567 average throughput is 2.19 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:56:08,982 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:08,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:08,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:08,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:08,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:08,989 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:08,989 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:08,989 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:08,989 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:08,990 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:08,990 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:08,991 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,991 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:08,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:08,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:08,992 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:08,992 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:08,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:08,994 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:08,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:08,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:08,994 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:08,994 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:08,994 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:08,995 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:08,995 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:08,995 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:08,995 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:08,995 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:08,995 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:08,995 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:08,995 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c8127616cf5355dd7f1db99764192b95 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-09 09:56:08,995 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=A 2023-04-09 09:56:08,995 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:56:08,995 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=B 2023-04-09 09:56:08,996 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:56:08,996 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=C 2023-04-09 09:56:08,996 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:56:09,019 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230409b2d2a87e8b5b477d87ec8a834ffa94fe to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409b2d2a87e8b5b477d87ec8a834ffa94fe 2023-04-09 09:56:09,025 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=154, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/A/d3891a5d69534554b00b403ba504d73f 2023-04-09 09:56:09,037 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=154 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/5bda5149d98740a18addd7a26ecca890 2023-04-09 09:56:09,047 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=154 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/C/9bde71d48cd34d158104f08f38f88121 2023-04-09 09:56:09,051 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/A/d3891a5d69534554b00b403ba504d73f as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/d3891a5d69534554b00b403ba504d73f 2023-04-09 09:56:09,055 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/d3891a5d69534554b00b403ba504d73f, entries=100, sequenceid=154, filesize=18.5 K 2023-04-09 09:56:09,056 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/5bda5149d98740a18addd7a26ecca890 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/5bda5149d98740a18addd7a26ecca890 2023-04-09 09:56:09,060 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/5bda5149d98740a18addd7a26ecca890, entries=100, sequenceid=154, filesize=9.4 K 2023-04-09 09:56:09,060 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/C/9bde71d48cd34d158104f08f38f88121 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/9bde71d48cd34d158104f08f38f88121 2023-04-09 09:56:09,065 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/9bde71d48cd34d158104f08f38f88121, entries=100, sequenceid=154, filesize=9.4 K 2023-04-09 09:56:09,065 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=0 B/0 for c8127616cf5355dd7f1db99764192b95 in 70ms, sequenceid=154, compaction requested=true 2023-04-09 09:56:09,066 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c8127616cf5355dd7f1db99764192b95: 2023-04-09 09:56:09,066 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:09,066 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:09,066 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:09,066 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,066 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:09,066 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:09,068 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:09,068 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:09,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:09,068 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:09,068 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:09,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:09,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:09,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:09,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:09,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:09,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7184df6b[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:09,070 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:09,070 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:09,071 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:09,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:09,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:09,071 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:09,072 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:09,072 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:09,072 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,072 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:09,073 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:09,073 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:09,073 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,073 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:09,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:09,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:09,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:09,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:09,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:09,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:09,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:09,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:09,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:09,076 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:09,076 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:09,076 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:09,076 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:09,076 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:09,077 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:09,077 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:09,077 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:09,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:09,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:09,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:09,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:09,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:09,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:09,078 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:09,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:09,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:09,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:09,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:09,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:09,079 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:09,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:09,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,082 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,082 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:09,082 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:09,082 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,082 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:09,082 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:56:09,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:09,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:09,082 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:09,083 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:09,083 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:09,082 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:09,083 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:09,083 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:09,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:09,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:09,083 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:09,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:09,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:09,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on c8127616cf5355dd7f1db99764192b95 2023-04-09 09:56:09,182 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c8127616cf5355dd7f1db99764192b95 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-09 09:56:09,182 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=A 2023-04-09 09:56:09,182 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:56:09,183 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=B 2023-04-09 09:56:09,183 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:56:09,183 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=C 2023-04-09 09:56:09,183 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:56:09,183 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:09,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:09,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:09,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:09,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:09,192 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:09,192 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:09,192 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:09,192 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:09,193 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:09,193 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:09,194 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:09,194 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:09,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:09,194 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,194 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:09,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:09,195 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:09,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:09,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:09,195 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:09,196 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:09,196 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:09,196 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:09,196 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:09,196 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:09,197 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:09,197 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:09,197 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:09,197 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:09,197 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:09,197 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:09,197 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:09,197 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:09,197 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,200 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:09,200 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:09,201 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,201 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:09,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:09,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:09,202 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:09,202 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:09,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:09,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:09,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:09,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:09,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:09,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d5c1293[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:09,204 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:09,204 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:09,206 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:09,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:09,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:09,206 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:09,206 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:09,206 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:09,208 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230409d9d45b4abe224a52a4c1667eee8003aa to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409d9d45b4abe224a52a4c1667eee8003aa 2023-04-09 09:56:09,209 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:09,209 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:09,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:09,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:09,209 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,209 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:09,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:09,209 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:09,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:09,210 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:09,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:09,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:09,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,213 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034169195, max timestamp=1681034169211Num uniques 859; 2023-04-09 09:56:09,213 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:56:09,213 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034169195, max timestamp=1681034169211Num uniques 859; 2023-04-09 09:56:09,213 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:56:09,213 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034169195, max timestamp=1681034169211Num uniques 859; 2023-04-09 09:56:09,214 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:56:09,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:09,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:09,214 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:09,214 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:09,214 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:09,214 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:09,214 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:09,215 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:09,216 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:09,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:09,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:09,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:09,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:09,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:09,216 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:09,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:09,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:09,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:09,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:09,217 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:09,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:09,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:09,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,219 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:09,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:09,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,223 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,223 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:09,223 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:09,223 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:09,223 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,224 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:09,224 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:09,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:09,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:09,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:09,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:09,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:09,224 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:09,224 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:09,224 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:09,224 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:09,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:09,224 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:09,237 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:09,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60436 deadline: 1681034229232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,237 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:09,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60448 deadline: 1681034229233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,237 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:09,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60506 deadline: 1681034229234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,239 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:09,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60460 deadline: 1681034229235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,239 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:09,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60498 deadline: 1681034229236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,309 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/5e41c9fa189642dbade8cc63d4ecf3a0 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/5e41c9fa189642dbade8cc63d4ecf3a0 2023-04-09 09:56:09,313 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c8127616cf5355dd7f1db99764192b95/B of c8127616cf5355dd7f1db99764192b95 into 5e41c9fa189642dbade8cc63d4ecf3a0(size=11.9 K), total size for store is 21.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:56:09,313 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c8127616cf5355dd7f1db99764192b95: 2023-04-09 09:56:09,313 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95., storeName=c8127616cf5355dd7f1db99764192b95/B, priority=12, startTime=1681034168878; duration=0sec 2023-04-09 09:56:09,313 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:56:09,314 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-09 09:56:09,315 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 57336 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-09 09:56:09,315 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1912): c8127616cf5355dd7f1db99764192b95/C is initiating minor compaction (all files) 2023-04-09 09:56:09,315 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c8127616cf5355dd7f1db99764192b95/C in TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:09,315 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/74b917e760f448caa7aad394f8a80885, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/c8b220063d05497ca6ca47ea08124297, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/d88ac10f0b5a4045a38d2e5441f5c8f0, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/9e49d1dbf7ca4ffc87d8cbfaa3cf445e, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/9bde71d48cd34d158104f08f38f88121] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp, totalSize=56.0 K 2023-04-09 09:56:09,316 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 74b917e760f448caa7aad394f8a80885, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1681034165658 2023-04-09 09:56:09,317 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting c8b220063d05497ca6ca47ea08124297, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1681034165739 2023-04-09 09:56:09,317 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting d88ac10f0b5a4045a38d2e5441f5c8f0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1681034165749 2023-04-09 09:56:09,318 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 9e49d1dbf7ca4ffc87d8cbfaa3cf445e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1681034167895 2023-04-09 09:56:09,318 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 9bde71d48cd34d158104f08f38f88121, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=154, earliestPutTs=1681034168016 2023-04-09 09:56:09,324 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:09,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:09,326 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] throttle.PressureAwareThroughputController(145): c8127616cf5355dd7f1db99764192b95#C#compaction#573 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:56:09,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:09,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:09,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:09,329 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:09,329 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:09,329 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:09,329 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:09,330 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:09,330 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:09,332 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,332 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:09,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:09,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:09,333 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:09,333 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:09,333 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:09,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:09,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:09,333 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:09,336 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:09,336 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:09,336 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:09,336 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:09,336 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:09,340 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:09,340 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:09,340 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:09,340 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:09,340 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:09,341 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:09,341 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:09,341 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:09,341 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,341 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:09,341 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:09,341 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:09,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60448 deadline: 1681034229338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,342 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:09,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60436 deadline: 1681034229338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,342 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:09,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60506 deadline: 1681034229339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,342 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:09,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60460 deadline: 1681034229340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,342 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:09,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60498 deadline: 1681034229341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,343 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,343 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:09,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:09,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:09,344 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:09,344 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:09,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:09,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:09,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:09,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:09,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:09,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a2a0261[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:09,346 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:09,346 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:09,347 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:09,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:09,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:09,347 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:09,347 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:09,347 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:09,347 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,347 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:09,349 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:09,349 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:09,349 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:09,349 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:09,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:09,350 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/C/8e798f5fa9b5494a8e5e449294b29eae as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/8e798f5fa9b5494a8e5e449294b29eae 2023-04-09 09:56:09,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:09,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:09,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:09,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:09,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:09,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,356 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in c8127616cf5355dd7f1db99764192b95/C of c8127616cf5355dd7f1db99764192b95 into 8e798f5fa9b5494a8e5e449294b29eae(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:56:09,356 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c8127616cf5355dd7f1db99764192b95: 2023-04-09 09:56:09,356 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95., storeName=c8127616cf5355dd7f1db99764192b95/C, priority=11, startTime=1681034168878; duration=0sec 2023-04-09 09:56:09,356 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:56:09,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:09,357 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:09,357 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:09,357 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:09,357 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:09,357 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:09,357 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:09,360 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:09,360 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:09,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:09,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:09,360 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:09,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:09,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:09,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:09,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:09,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:09,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:09,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:09,361 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:09,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:09,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:09,362 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:09,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:09,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,364 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:09,364 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:09,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:09,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:09,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:09,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:09,365 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,365 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:09,365 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:09,365 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:09,365 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,365 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:09,365 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:09,365 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:09,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:09,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:09,365 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:09,365 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:09,465 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:09,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:09,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:09,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:09,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:09,471 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:09,471 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:09,471 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:09,471 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:09,471 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:09,471 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:09,472 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,473 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:09,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:09,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:09,473 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,473 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:09,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:09,474 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:09,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:09,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:09,474 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:09,479 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:09,479 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:09,479 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:09,479 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:09,479 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:09,480 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:09,480 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:09,480 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:09,480 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:09,480 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:09,480 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:09,480 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:09,480 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:09,480 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,480 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:09,480 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:09,482 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:09,482 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:09,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:09,482 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:09,482 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:09,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:09,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:09,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:09,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:09,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:09,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@267a3464[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:09,483 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:09,483 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:09,485 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:09,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:09,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:09,485 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:09,486 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:09,486 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:09,486 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,486 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:09,489 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:09,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:09,489 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:09,489 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:09,489 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:09,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:09,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:09,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:09,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:09,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:09,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:09,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:09,492 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:09,492 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:09,492 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:09,492 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:09,492 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:09,494 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:09,494 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:09,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:09,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:09,495 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:09,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:09,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:09,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:09,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:09,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:09,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:09,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:09,496 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:09,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:09,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:09,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,497 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:09,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:09,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,499 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,499 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:09,499 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:09,499 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,499 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:09,500 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:09,500 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:09,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:09,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:09,501 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:09,501 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:09,501 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:09,501 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:56:09,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:09,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:09,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:09,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:09,501 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:09,501 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:09,545 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:09,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60448 deadline: 1681034229543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,546 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:09,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60498 deadline: 1681034229543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,546 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:09,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60460 deadline: 1681034229544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,549 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:09,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60506 deadline: 1681034229547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,550 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:09,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60436 deadline: 1681034229548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,602 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:09,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:09,603 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:09,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:09,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:09,605 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:09,605 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:09,605 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:09,605 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:09,605 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:09,605 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:09,611 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,611 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:09,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:09,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:09,612 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,612 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:09,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:09,612 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:09,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:09,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:09,612 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:09,613 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:09,613 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:09,613 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:09,613 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:09,613 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:09,613 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:09,613 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:09,613 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:09,613 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:09,613 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:09,613 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:09,613 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:09,613 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:09,613 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,613 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:09,613 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:09,614 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,614 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:09,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:09,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:09,615 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:09,615 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:09,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:09,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:09,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:09,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:09,616 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:09,616 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71d2d847[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:09,616 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:09,616 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:09,617 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:09,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:09,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:09,617 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:09,617 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:09,617 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:09,617 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,617 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:09,618 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,618 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:09,618 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:09,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:09,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:09,618 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:09,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:09,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:09,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:09,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:09,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:09,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:09,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:09,620 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:09,621 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:09,621 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:09,621 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:09,621 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:09,623 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:09,623 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:09,623 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:09,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:09,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:09,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:09,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:09,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:09,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:09,624 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:09,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:09,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:09,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:09,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:09,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:09,626 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:09,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:09,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,630 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,630 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:09,630 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:09,630 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,630 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:09,630 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:09,630 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:09,630 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:09,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:09,630 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:56:09,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:09,630 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:09,630 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:09,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:09,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:09,631 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:09,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:09,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:09,631 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:09,634 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=164, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/A/d795c2cedb8943afaad609b09ea6dfcb 2023-04-09 09:56:09,642 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=164 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/d47f960990664f22ab9c70dae55d83c9 2023-04-09 09:56:09,651 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=164 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/C/297f6d7fced140c9ac8a7d2a705da177 2023-04-09 09:56:09,655 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/A/d795c2cedb8943afaad609b09ea6dfcb as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/d795c2cedb8943afaad609b09ea6dfcb 2023-04-09 09:56:09,661 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/d795c2cedb8943afaad609b09ea6dfcb, entries=350, sequenceid=164, filesize=52.6 K 2023-04-09 09:56:09,662 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/d47f960990664f22ab9c70dae55d83c9 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/d47f960990664f22ab9c70dae55d83c9 2023-04-09 09:56:09,665 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/d47f960990664f22ab9c70dae55d83c9, entries=150, sequenceid=164, filesize=11.8 K 2023-04-09 09:56:09,666 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/C/297f6d7fced140c9ac8a7d2a705da177 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/297f6d7fced140c9ac8a7d2a705da177 2023-04-09 09:56:09,669 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/297f6d7fced140c9ac8a7d2a705da177, entries=150, sequenceid=164, filesize=11.8 K 2023-04-09 09:56:09,670 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for c8127616cf5355dd7f1db99764192b95 in 488ms, sequenceid=164, compaction requested=true 2023-04-09 09:56:09,670 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c8127616cf5355dd7f1db99764192b95: 2023-04-09 09:56:09,670 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:56:09,670 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:56:09,670 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-09 09:56:09,670 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 4 compacting, 2 eligible, 16 blocking 2023-04-09 09:56:09,670 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-09 09:56:09,671 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-09 09:56:09,671 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. because compaction request was cancelled 2023-04-09 09:56:09,671 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-09 09:56:09,678 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33852 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-09 09:56:09,678 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1912): c8127616cf5355dd7f1db99764192b95/B is initiating minor compaction (all files) 2023-04-09 09:56:09,678 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c8127616cf5355dd7f1db99764192b95/B in TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:09,679 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/5e41c9fa189642dbade8cc63d4ecf3a0, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/5bda5149d98740a18addd7a26ecca890, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/d47f960990664f22ab9c70dae55d83c9] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp, totalSize=33.1 K 2023-04-09 09:56:09,679 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 5e41c9fa189642dbade8cc63d4ecf3a0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1681034167895 2023-04-09 09:56:09,679 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 5bda5149d98740a18addd7a26ecca890, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=154, earliestPutTs=1681034168016 2023-04-09 09:56:09,679 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting d47f960990664f22ab9c70dae55d83c9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=164, earliestPutTs=1681034169170 2023-04-09 09:56:09,686 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] throttle.PressureAwareThroughputController(145): c8127616cf5355dd7f1db99764192b95#B#compaction#576 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:56:09,731 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:09,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:09,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:09,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:09,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:09,734 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:09,734 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:09,734 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:09,734 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:09,735 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:09,735 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:09,736 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:09,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:09,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:09,736 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,737 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,737 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:09,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:09,738 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:09,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:09,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:09,738 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:09,740 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:09,740 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:09,740 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:09,740 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:09,740 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:09,741 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:09,741 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:09,741 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:09,741 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:09,741 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c8127616cf5355dd7f1db99764192b95 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-09 09:56:09,741 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=A 2023-04-09 09:56:09,742 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:56:09,742 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=B 2023-04-09 09:56:09,742 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:56:09,742 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=C 2023-04-09 09:56:09,742 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:56:09,751 WARN [DataStreamer for file /user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230409dde041b0a55442b1b647c1463108bbc4] hdfs.DataStreamer(982): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) at java.lang.Thread.join(Thread.java:1331) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:630) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:807) 2023-04-09 09:56:09,754 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230409dde041b0a55442b1b647c1463108bbc4 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409dde041b0a55442b1b647c1463108bbc4 2023-04-09 09:56:09,760 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=203, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/A/033c11eb33fa4911a6ab53bc55dbac9d 2023-04-09 09:56:09,774 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/686c3ad2a7754a5e909b87deed1874ff 2023-04-09 09:56:09,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on c8127616cf5355dd7f1db99764192b95 2023-04-09 09:56:09,853 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:09,862 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:09,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60506 deadline: 1681034229856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,865 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:09,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60436 deadline: 1681034229860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,866 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:09,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60498 deadline: 1681034229861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,867 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:09,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60448 deadline: 1681034229862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,867 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:09,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60460 deadline: 1681034229862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,965 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:09,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60506 deadline: 1681034229963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,968 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:09,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60436 deadline: 1681034229966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,970 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:09,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60498 deadline: 1681034229968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,970 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:09,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60448 deadline: 1681034229968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:09,970 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:09,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60460 deadline: 1681034229968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:10,106 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/ca62cc483bc247ae9bbbfb85e87a797d as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/ca62cc483bc247ae9bbbfb85e87a797d 2023-04-09 09:56:10,110 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c8127616cf5355dd7f1db99764192b95/B of c8127616cf5355dd7f1db99764192b95 into ca62cc483bc247ae9bbbfb85e87a797d(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:56:10,111 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c8127616cf5355dd7f1db99764192b95: 2023-04-09 09:56:10,111 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95., storeName=c8127616cf5355dd7f1db99764192b95/B, priority=13, startTime=1681034169670; duration=0sec 2023-04-09 09:56:10,111 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:56:10,111 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-09 09:56:10,111 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-09 09:56:10,112 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-09 09:56:10,112 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. because compaction request was cancelled 2023-04-09 09:56:10,150 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/A/aed1071f46a94f9b82eacbf3a28f23fc as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/aed1071f46a94f9b82eacbf3a28f23fc 2023-04-09 09:56:10,156 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c8127616cf5355dd7f1db99764192b95/A of c8127616cf5355dd7f1db99764192b95 into aed1071f46a94f9b82eacbf3a28f23fc(size=25.4 K), total size for store is 96.5 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-09 09:56:10,156 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c8127616cf5355dd7f1db99764192b95: 2023-04-09 09:56:10,156 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95., storeName=c8127616cf5355dd7f1db99764192b95/A, priority=12, startTime=1681034168878; duration=1sec 2023-04-09 09:56:10,157 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:56:10,170 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:10,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60506 deadline: 1681034230167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:10,171 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:10,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60436 deadline: 1681034230169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:10,173 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:10,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60498 deadline: 1681034230171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:10,175 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:10,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60460 deadline: 1681034230172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:10,175 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:10,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60448 deadline: 1681034230172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:10,182 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/C/e4312df7fdb04d74b6a119156a1e1685 2023-04-09 09:56:10,187 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/A/033c11eb33fa4911a6ab53bc55dbac9d as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/033c11eb33fa4911a6ab53bc55dbac9d 2023-04-09 09:56:10,191 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/033c11eb33fa4911a6ab53bc55dbac9d, entries=150, sequenceid=203, filesize=25.3 K 2023-04-09 09:56:10,191 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/686c3ad2a7754a5e909b87deed1874ff as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/686c3ad2a7754a5e909b87deed1874ff 2023-04-09 09:56:10,195 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/686c3ad2a7754a5e909b87deed1874ff, entries=150, sequenceid=203, filesize=11.8 K 2023-04-09 09:56:10,197 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/C/e4312df7fdb04d74b6a119156a1e1685 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/e4312df7fdb04d74b6a119156a1e1685 2023-04-09 09:56:10,206 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/e4312df7fdb04d74b6a119156a1e1685, entries=150, sequenceid=203, filesize=11.8 K 2023-04-09 09:56:10,207 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=53.67 KB/54960 for c8127616cf5355dd7f1db99764192b95 in 466ms, sequenceid=203, compaction requested=true 2023-04-09 09:56:10,207 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c8127616cf5355dd7f1db99764192b95: 2023-04-09 09:56:10,207 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:10,207 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:10,208 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:10,208 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:10,208 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:10,208 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:10,210 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:10,211 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:10,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:10,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:10,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:10,211 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:10,211 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:10,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:10,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:10,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:10,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:10,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:10,213 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:10,213 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ce79464[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:10,213 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:10,213 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:10,214 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:10,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:10,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:10,214 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:10,214 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:10,214 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:10,214 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:10,214 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:10,216 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:10,217 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:10,217 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:10,217 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:10,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:10,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:10,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:10,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:10,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:10,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:10,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:10,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:10,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:10,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:10,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:10,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:10,220 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:10,220 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:10,220 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:10,221 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:10,221 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:10,222 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:10,222 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:10,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:10,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:10,222 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:10,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:10,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:10,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:10,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:10,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:10,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:10,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:10,224 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:10,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:10,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:10,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:10,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:10,225 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:10,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:10,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:10,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:10,229 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:10,229 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:10,229 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:10,230 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:10,230 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:10,230 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:10,230 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:10,230 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:10,230 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:10,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:10,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:10,230 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:10,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:10,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:10,230 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:10,230 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:10,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:10,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:10,330 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:10,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:10,333 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:10,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:10,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:10,334 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:10,334 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:10,334 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:10,334 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:10,334 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:10,335 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:10,336 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:10,336 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:10,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:10,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:10,336 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:10,336 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:10,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:10,337 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:10,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:10,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:10,337 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:10,340 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:10,340 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:10,340 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:10,340 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:10,340 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:10,344 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:10,344 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:10,344 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:10,344 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:10,344 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c8127616cf5355dd7f1db99764192b95 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-09 09:56:10,344 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=A 2023-04-09 09:56:10,344 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:56:10,344 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=B 2023-04-09 09:56:10,344 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:56:10,345 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=C 2023-04-09 09:56:10,345 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:56:10,372 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040971157784980d4e34ae71acad650e9862 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040971157784980d4e34ae71acad650e9862 2023-04-09 09:56:10,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on c8127616cf5355dd7f1db99764192b95 2023-04-09 09:56:10,476 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:10,497 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1681034170484, max timestamp=1681034170495Num uniques 859; 2023-04-09 09:56:10,497 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1681034170484, max timestamp=1681034170495Num uniques 859; 2023-04-09 09:56:10,497 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-09 09:56:10,497 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1681034170484, max timestamp=1681034170495Num uniques 859; 2023-04-09 09:56:10,497 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-09 09:56:10,497 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-09 09:56:10,507 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:10,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60448 deadline: 1681034230503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:10,509 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:10,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60436 deadline: 1681034230506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:10,510 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:10,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60498 deadline: 1681034230506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:10,510 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:10,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60460 deadline: 1681034230506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:10,510 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:10,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60506 deadline: 1681034230507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:10,613 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:10,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60460 deadline: 1681034230612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:10,614 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:10,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60498 deadline: 1681034230612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:10,614 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:10,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60436 deadline: 1681034230612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:10,614 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:10,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60448 deadline: 1681034230612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:10,615 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:10,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60506 deadline: 1681034230612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:10,777 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=216, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/A/38eaff8221ad4f91a144403286f0080a 2023-04-09 09:56:10,802 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=216 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/65e557d0523d4babb81e58d469ef8ed5 2023-04-09 09:56:10,812 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=216 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/C/926a1d5785144b4783b1224640ff7581 2023-04-09 09:56:10,815 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:10,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60460 deadline: 1681034230815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:10,816 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:10,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60498 deadline: 1681034230816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:10,816 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:10,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60448 deadline: 1681034230816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:10,820 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:10,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60506 deadline: 1681034230818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:10,820 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:10,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60436 deadline: 1681034230818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:10,821 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/A/38eaff8221ad4f91a144403286f0080a as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/38eaff8221ad4f91a144403286f0080a 2023-04-09 09:56:10,824 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/38eaff8221ad4f91a144403286f0080a, entries=150, sequenceid=216, filesize=25.3 K 2023-04-09 09:56:10,826 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/65e557d0523d4babb81e58d469ef8ed5 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/65e557d0523d4babb81e58d469ef8ed5 2023-04-09 09:56:10,830 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/65e557d0523d4babb81e58d469ef8ed5, entries=150, sequenceid=216, filesize=11.8 K 2023-04-09 09:56:10,831 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/C/926a1d5785144b4783b1224640ff7581 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/926a1d5785144b4783b1224640ff7581 2023-04-09 09:56:10,841 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/926a1d5785144b4783b1224640ff7581, entries=150, sequenceid=216, filesize=11.8 K 2023-04-09 09:56:10,844 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for c8127616cf5355dd7f1db99764192b95 in 500ms, sequenceid=216, compaction requested=true 2023-04-09 09:56:10,845 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c8127616cf5355dd7f1db99764192b95: 2023-04-09 09:56:10,845 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:10,845 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:10,845 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:10,845 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:10,845 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:10,845 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:10,846 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:10,846 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:10,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:10,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:10,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:10,847 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:10,847 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:10,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:10,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:10,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:10,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:10,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:10,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:10,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76a4b693[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:10,849 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:10,850 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:10,851 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:10,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:10,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:10,851 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:10,851 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:10,851 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:10,851 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:10,851 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:10,854 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:10,854 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:10,854 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:10,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:10,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:10,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:10,854 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:10,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:10,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:10,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:10,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:10,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:10,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:10,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:10,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:10,856 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:10,857 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:10,857 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:10,857 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:10,857 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:10,857 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:10,858 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:10,858 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:10,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:10,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:10,858 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:10,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:10,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:10,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:10,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:10,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:10,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:10,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:10,859 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:10,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:10,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:10,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:10,861 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:10,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:10,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:10,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:10,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:10,863 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:10,863 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:10,863 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:10,863 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:10,863 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:10,863 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:10,863 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:10,863 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:10,863 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:10,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:10,864 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:10,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:10,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:10,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:10,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:10,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:10,864 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:10,864 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:10,964 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:10,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:10,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:10,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:10,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:10,968 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:10,968 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:10,968 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:10,968 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:10,969 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:10,969 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:10,970 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:10,970 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:10,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:10,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:10,970 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:10,970 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:10,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:10,971 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:10,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:10,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:10,971 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:10,980 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:10,980 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:10,980 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:10,980 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:10,980 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:10,992 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:10,992 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:10,996 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:10,996 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:10,996 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c8127616cf5355dd7f1db99764192b95 3/3 column families, dataSize=134.18 KB heapSize=366.23 KB 2023-04-09 09:56:10,996 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=A 2023-04-09 09:56:10,996 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:56:10,996 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=B 2023-04-09 09:56:10,997 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:56:10,997 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=C 2023-04-09 09:56:10,997 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:56:11,062 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304091660084efe574ada95722cf9e78850f6 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304091660084efe574ada95722cf9e78850f6 2023-04-09 09:56:11,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on c8127616cf5355dd7f1db99764192b95 2023-04-09 09:56:11,119 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:11,129 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:11,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60448 deadline: 1681034231125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:11,133 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:11,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60498 deadline: 1681034231128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:11,133 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:11,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60506 deadline: 1681034231128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:11,134 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:11,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60436 deadline: 1681034231129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:11,134 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:11,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60460 deadline: 1681034231129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:11,232 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:11,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60448 deadline: 1681034231230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:11,237 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:11,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60498 deadline: 1681034231234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:11,238 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:11,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60506 deadline: 1681034231234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:11,238 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:11,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60460 deadline: 1681034231235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:11,238 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:11,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60436 deadline: 1681034231235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:11,437 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:11,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60448 deadline: 1681034231434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:11,441 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:11,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60498 deadline: 1681034231438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:11,442 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:11,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60460 deadline: 1681034231439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:11,443 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:11,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60506 deadline: 1681034231440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:11,443 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:11,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60436 deadline: 1681034231440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:11,482 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=249, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/A/e25930902a7346a5a1ac05688b0fc3ae 2023-04-09 09:56:11,744 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:11,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60448 deadline: 1681034231741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:11,745 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:11,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60498 deadline: 1681034231745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:11,745 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:11,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60460 deadline: 1681034231745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:11,746 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:11,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60436 deadline: 1681034231745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:11,749 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:11,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60506 deadline: 1681034231746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:11,918 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=249 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/74721bdf15624468891165c7f5afdb83 2023-04-09 09:56:12,249 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:12,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60436 deadline: 1681034232247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:12,250 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:12,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60498 deadline: 1681034232248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:12,250 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:12,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60448 deadline: 1681034232248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:12,252 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:12,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60460 deadline: 1681034232250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:12,254 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:12,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60506 deadline: 1681034232251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:12,330 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=249 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/C/12fe1a6a4b79499ca7558b3e1cf5421a 2023-04-09 09:56:12,335 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/A/e25930902a7346a5a1ac05688b0fc3ae as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/e25930902a7346a5a1ac05688b0fc3ae 2023-04-09 09:56:12,339 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/e25930902a7346a5a1ac05688b0fc3ae, entries=150, sequenceid=249, filesize=25.3 K 2023-04-09 09:56:12,340 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/74721bdf15624468891165c7f5afdb83 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/74721bdf15624468891165c7f5afdb83 2023-04-09 09:56:12,344 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/74721bdf15624468891165c7f5afdb83, entries=150, sequenceid=249, filesize=11.8 K 2023-04-09 09:56:12,345 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/C/12fe1a6a4b79499ca7558b3e1cf5421a as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/12fe1a6a4b79499ca7558b3e1cf5421a 2023-04-09 09:56:12,360 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/12fe1a6a4b79499ca7558b3e1cf5421a, entries=150, sequenceid=249, filesize=11.8 K 2023-04-09 09:56:12,361 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~366.19 KB/374976, currentSize=60.38 KB/61830 for c8127616cf5355dd7f1db99764192b95 in 1365ms, sequenceid=249, compaction requested=true 2023-04-09 09:56:12,361 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c8127616cf5355dd7f1db99764192b95: 2023-04-09 09:56:12,361 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:12,361 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:12,361 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:12,361 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:12,364 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:12,364 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:12,365 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:12,365 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:12,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:12,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:12,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:12,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:12,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:12,366 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:12,367 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:12,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:12,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:12,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:12,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:12,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10a6537[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:12,367 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:12,368 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:12,369 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:12,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:12,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:12,369 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:12,369 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:12,369 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:12,369 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:12,369 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:12,372 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:12,372 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:12,372 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:12,372 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:12,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:12,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:12,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:12,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:12,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:12,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:12,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:12,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:12,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:12,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:12,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:12,377 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:12,377 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:12,377 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:12,377 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:12,377 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:12,377 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:12,379 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:12,379 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:12,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:12,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:12,379 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:12,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:12,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:12,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:12,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:12,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:12,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:12,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:12,380 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:12,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:12,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:12,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:12,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:12,382 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:12,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:12,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:12,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:12,385 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:12,385 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:12,385 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:12,385 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:12,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:12,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:12,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:12,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:12,385 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:12,386 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:12,386 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:12,386 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:12,386 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:12,386 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:12,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:12,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:12,386 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:12,386 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:12,486 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:12,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:12,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:12,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:12,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:12,496 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:12,496 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:12,496 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:12,496 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:12,496 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:12,496 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:12,498 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:12,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:12,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:12,498 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:12,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:12,498 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:12,498 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:12,498 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:12,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:12,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:12,499 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:12,504 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:12,504 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:12,504 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:12,504 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:12,504 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:12,512 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:12,512 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:12,524 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:12,524 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:12,524 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c8127616cf5355dd7f1db99764192b95 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-09 09:56:12,524 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=A 2023-04-09 09:56:12,524 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:56:12,524 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=B 2023-04-09 09:56:12,524 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:56:12,524 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=C 2023-04-09 09:56:12,524 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:56:12,981 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304090fce5e7cf9f2491f9346e94e9bfdd740 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304090fce5e7cf9f2491f9346e94e9bfdd740 2023-04-09 09:56:12,987 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=261, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/A/22c905efc1c744049ad4d0d8cb342618 2023-04-09 09:56:12,998 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=261 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/2fd097b323724390ac79ca324ff52ac4 2023-04-09 09:56:13,044 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=261 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/C/00a3eb90e67d4fa9adb2e9841012c4e0 2023-04-09 09:56:13,049 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/A/22c905efc1c744049ad4d0d8cb342618 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/22c905efc1c744049ad4d0d8cb342618 2023-04-09 09:56:13,055 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/22c905efc1c744049ad4d0d8cb342618, entries=150, sequenceid=261, filesize=25.4 K 2023-04-09 09:56:13,056 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/2fd097b323724390ac79ca324ff52ac4 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/2fd097b323724390ac79ca324ff52ac4 2023-04-09 09:56:13,059 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/2fd097b323724390ac79ca324ff52ac4, entries=150, sequenceid=261, filesize=11.9 K 2023-04-09 09:56:13,059 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/C/00a3eb90e67d4fa9adb2e9841012c4e0 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/00a3eb90e67d4fa9adb2e9841012c4e0 2023-04-09 09:56:13,075 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/00a3eb90e67d4fa9adb2e9841012c4e0, entries=150, sequenceid=261, filesize=11.9 K 2023-04-09 09:56:13,076 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=0 B/0 for c8127616cf5355dd7f1db99764192b95 in 552ms, sequenceid=261, compaction requested=true 2023-04-09 09:56:13,076 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c8127616cf5355dd7f1db99764192b95: 2023-04-09 09:56:13,076 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:13,076 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:13,077 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:13,077 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,077 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:13,077 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:13,081 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,081 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:13,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:13,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:13,082 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:13,082 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:13,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:13,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:13,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:13,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:13,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:13,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66fd85b1[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:13,083 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:13,083 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:13,084 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:13,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:13,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:13,084 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:13,085 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:13,085 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,085 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:13,085 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:13,086 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,086 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:13,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:13,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:13,086 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:13,086 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:13,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:13,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:13,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:13,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:13,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:13,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:13,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:13,089 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:13,089 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:13,089 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:13,089 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:13,089 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:13,090 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:13,090 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:13,090 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:13,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:13,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:13,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:13,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:13,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:13,091 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:13,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:13,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:13,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:13,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:13,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:13,092 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:13,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:13,095 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,095 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:13,095 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:13,095 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,095 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:13,095 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:13,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:13,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:13,095 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:13,095 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:13,096 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:56:13,096 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:13,096 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:13,100 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:13,100 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:13,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:13,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:13,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:13,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:13,196 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:13,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:13,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:13,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:13,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:13,207 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:13,207 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:13,207 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:13,207 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:13,207 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:13,207 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:13,209 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:13,209 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:13,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:13,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:13,210 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,210 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:13,210 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:13,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:13,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:13,210 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:13,210 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:13,211 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:13,211 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:13,211 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:13,211 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:13,211 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:13,211 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:13,211 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:13,211 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:13,211 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c8127616cf5355dd7f1db99764192b95: 2023-04-09 09:56:13,211 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:13,211 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:13,212 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:13,212 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,212 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:13,212 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:13,213 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:13,213 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:13,213 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:13,215 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:13,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:13,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:13,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:13,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:13,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:13,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:13,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@560979be[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:13,217 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:13,217 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:13,218 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:13,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:13,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:13,218 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:13,218 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:13,218 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:13,218 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,218 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:13,221 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:13,221 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:13,221 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:13,221 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:13,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:13,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:13,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:13,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:13,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:13,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:13,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:13,223 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:13,223 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:13,223 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:13,223 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:13,223 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:13,224 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:13,225 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:13,225 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:13,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:13,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:13,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:13,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:13,225 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:13,225 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:13,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:13,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:13,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:13,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:13,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:13,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:13,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:13,227 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:13,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:13,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,229 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:13,229 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:13,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:13,229 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,229 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:13,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:13,229 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:13,229 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:56:13,229 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:13,229 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,229 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:13,229 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:13,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:13,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:13,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:13,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:13,229 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:13,230 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:13,230 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:13,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on c8127616cf5355dd7f1db99764192b95 2023-04-09 09:56:13,284 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c8127616cf5355dd7f1db99764192b95 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-09 09:56:13,284 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=A 2023-04-09 09:56:13,284 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:56:13,284 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=B 2023-04-09 09:56:13,284 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:56:13,284 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=C 2023-04-09 09:56:13,284 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:56:13,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1681034173305, max timestamp=1681034173314Num uniques 859; 2023-04-09 09:56:13,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1681034173305, max timestamp=1681034173314Num uniques 859; 2023-04-09 09:56:13,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-09 09:56:13,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-09 09:56:13,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1681034173305, max timestamp=1681034173314Num uniques 859; 2023-04-09 09:56:13,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-09 09:56:13,324 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304095b00c3ee0c2e48a69ea4e420fa9b49b0 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304095b00c3ee0c2e48a69ea4e420fa9b49b0 2023-04-09 09:56:13,330 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:13,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:13,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:13,353 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:13,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60436 deadline: 1681034233348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,358 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:13,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60506 deadline: 1681034233350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,359 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:13,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60498 deadline: 1681034233352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,359 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:13,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60460 deadline: 1681034233353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,360 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=272, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/A/19f6883ee25e49caaea691ed2bc7339d 2023-04-09 09:56:13,364 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:13,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60448 deadline: 1681034233360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:13,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:13,368 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:13,368 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:13,368 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:13,368 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:13,369 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:13,369 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:13,370 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,370 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:13,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:13,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:13,370 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,370 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:13,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:13,371 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:13,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:13,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:13,371 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:13,387 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:13,387 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:13,387 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:13,388 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:13,388 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:13,396 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:13,396 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:13,397 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:13,397 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:13,397 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:13,397 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:13,398 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:13,398 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:13,398 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,398 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:13,398 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:13,400 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,400 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:13,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:13,401 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:13,401 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:13,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:13,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:13,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:13,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:13,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:13,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:13,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ca5dc2[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:13,403 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:13,403 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:13,404 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:13,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:13,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:13,404 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:13,404 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:13,404 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:13,404 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,404 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:13,405 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:13,405 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:13,405 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:13,405 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:13,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:13,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:13,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:13,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:13,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:13,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:13,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:13,409 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:13,409 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:13,409 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:13,409 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:13,409 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:13,409 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:13,411 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:13,411 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:13,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:13,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:13,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:13,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:13,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:13,411 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:13,411 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:13,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:13,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:13,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:13,413 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,415 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:13,416 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:13,416 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:13,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:13,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:13,416 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,416 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:13,416 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:13,416 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,416 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:13,416 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:13,416 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:13,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:13,416 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:13,416 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:13,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:13,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:13,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:13,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:13,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:13,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:13,417 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:56:13,459 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:13,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60436 deadline: 1681034233455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,462 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:13,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60506 deadline: 1681034233459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,463 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:13,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60498 deadline: 1681034233460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,463 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:13,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60460 deadline: 1681034233460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,485 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:13,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60448 deadline: 1681034233466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,516 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:13,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:13,519 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:13,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:13,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:13,532 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:13,532 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:13,532 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:13,532 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:13,533 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:13,533 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:13,534 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,534 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:13,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:13,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:13,534 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:13,534 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:13,535 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:13,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:13,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:13,535 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:13,544 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:13,544 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:13,544 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:13,544 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:13,545 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:13,545 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:13,545 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:13,545 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:13,545 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:13,545 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:13,545 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:13,545 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:13,545 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,545 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:13,545 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:13,546 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,546 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:13,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:13,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:13,547 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:13,547 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:13,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:13,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:13,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:13,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:13,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:13,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@573ba391[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:13,549 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:13,549 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:13,556 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:13,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:13,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:13,556 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,557 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:13,557 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:13,557 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:13,557 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:13,558 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:13,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:13,559 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:13,559 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:13,559 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:13,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:13,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:13,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:13,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:13,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:13,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:13,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:13,562 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:13,562 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:13,562 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:13,562 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:13,562 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:13,563 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:13,563 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:13,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:13,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:13,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:13,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:13,563 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:13,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:13,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:13,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:13,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:13,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:13,565 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:13,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:13,567 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:13,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:13,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:13,570 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:13,570 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:13,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:13,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:13,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:13,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:13,571 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:13,571 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,571 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:13,571 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:13,571 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,571 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:13,571 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:13,571 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:13,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:13,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:13,571 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:13,571 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:13,572 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:56:13,666 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:13,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60460 deadline: 1681034233666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,670 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:13,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60436 deadline: 1681034233667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,671 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:13,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60506 deadline: 1681034233667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,671 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:13,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60498 deadline: 1681034233668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,671 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:13,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:13,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:13,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:13,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:13,680 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:13,680 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:13,680 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:13,680 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:13,680 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:13,681 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:13,683 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,683 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,683 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:13,688 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:13,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:13,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:13,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:13,689 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:13,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:13,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:13,689 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:13,690 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:13,690 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:13,690 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:13,690 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:13,690 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:13,695 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:13,695 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:13,695 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:13,696 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:13,696 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:13,696 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:13,696 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:13,696 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:13,696 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,696 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:13,696 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:13,698 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:13,698 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60448 deadline: 1681034233693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:13,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:13,698 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:13,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:13,698 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:13,699 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:13,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:13,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:13,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:13,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:13,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d6164ca[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:13,700 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:13,700 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:13,701 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:13,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:13,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:13,701 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:13,702 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:13,702 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:13,702 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,702 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:13,703 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:13,703 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:13,703 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:13,703 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:13,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:13,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:13,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:13,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:13,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:13,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:13,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:13,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:13,705 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:13,705 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:13,705 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:13,705 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:13,705 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:13,708 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:13,708 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:13,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:13,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:13,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:13,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:13,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:13,709 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,710 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:13,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:13,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:13,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:13,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:13,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:13,710 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:13,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:13,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:13,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:13,712 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,712 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:13,712 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:13,712 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,712 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:13,712 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:13,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:13,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:13,712 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:13,713 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:56:13,712 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:13,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:13,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:13,713 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:13,713 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:13,713 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:13,713 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:13,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:13,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:13,813 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:13,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:13,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:13,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:13,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:13,826 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:13,826 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:13,826 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:13,826 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:13,826 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=272 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/a2925f0d1a1a4d188f830cd86c0d5ca4 2023-04-09 09:56:13,826 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:13,826 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:13,828 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:13,828 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:13,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:13,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:13,828 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,828 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:13,828 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:13,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:13,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:13,829 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:13,844 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:13,844 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:13,844 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:13,844 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:13,844 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:13,852 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:13,852 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:13,853 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:13,853 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:13,853 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:13,853 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:13,853 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:13,853 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:13,853 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,853 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:13,853 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:13,854 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,854 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:13,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:13,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:13,855 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:13,855 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:13,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:13,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:13,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:13,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:13,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:13,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40382251[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:13,857 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:13,857 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:13,862 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=272 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/C/1d72fab54415439fbe236e8e25012ee6 2023-04-09 09:56:13,864 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:13,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:13,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:13,865 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:13,865 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:13,865 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:13,865 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,865 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:13,866 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,866 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:13,866 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:13,867 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:13,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:13,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:13,866 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/A/19f6883ee25e49caaea691ed2bc7339d as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/19f6883ee25e49caaea691ed2bc7339d 2023-04-09 09:56:13,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:13,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:13,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:13,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:13,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:13,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,871 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/19f6883ee25e49caaea691ed2bc7339d, entries=350, sequenceid=272, filesize=52.9 K 2023-04-09 09:56:13,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:13,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:13,872 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:13,873 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:13,873 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:13,873 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:13,873 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:13,873 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/a2925f0d1a1a4d188f830cd86c0d5ca4 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/a2925f0d1a1a4d188f830cd86c0d5ca4 2023-04-09 09:56:13,874 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:13,874 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:13,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:13,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:13,874 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:13,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:13,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:13,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:13,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:13,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:13,875 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:13,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:13,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:13,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:13,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:13,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,877 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,877 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/a2925f0d1a1a4d188f830cd86c0d5ca4, entries=150, sequenceid=272, filesize=11.9 K 2023-04-09 09:56:13,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:13,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:13,878 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/C/1d72fab54415439fbe236e8e25012ee6 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/1d72fab54415439fbe236e8e25012ee6 2023-04-09 09:56:13,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,880 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:13,880 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,880 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:13,880 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:13,880 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:13,880 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:13,880 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:13,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:13,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:13,880 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:56:13,880 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:13,880 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:13,880 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:13,881 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,881 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:13,881 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:13,881 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:13,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:13,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:13,882 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/1d72fab54415439fbe236e8e25012ee6, entries=150, sequenceid=272, filesize=11.9 K 2023-04-09 09:56:13,883 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for c8127616cf5355dd7f1db99764192b95 in 599ms, sequenceid=272, compaction requested=true 2023-04-09 09:56:13,883 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c8127616cf5355dd7f1db99764192b95: 2023-04-09 09:56:13,884 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-09 09:56:13,884 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:56:13,884 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:56:13,884 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:56:13,884 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-09 09:56:13,886 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 72756 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-09 09:56:13,886 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): c8127616cf5355dd7f1db99764192b95/B is initiating minor compaction (all files) 2023-04-09 09:56:13,886 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c8127616cf5355dd7f1db99764192b95/B in TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:13,886 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/ca62cc483bc247ae9bbbfb85e87a797d, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/686c3ad2a7754a5e909b87deed1874ff, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/65e557d0523d4babb81e58d469ef8ed5, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/74721bdf15624468891165c7f5afdb83, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/2fd097b323724390ac79ca324ff52ac4, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/a2925f0d1a1a4d188f830cd86c0d5ca4] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp, totalSize=71.1 K 2023-04-09 09:56:13,886 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 256740 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-09 09:56:13,886 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1912): c8127616cf5355dd7f1db99764192b95/A is initiating minor compaction (all files) 2023-04-09 09:56:13,887 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c8127616cf5355dd7f1db99764192b95/A in TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:13,887 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/aed1071f46a94f9b82eacbf3a28f23fc, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/d3891a5d69534554b00b403ba504d73f, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/d795c2cedb8943afaad609b09ea6dfcb, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/033c11eb33fa4911a6ab53bc55dbac9d, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/38eaff8221ad4f91a144403286f0080a, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/e25930902a7346a5a1ac05688b0fc3ae, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/22c905efc1c744049ad4d0d8cb342618, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/19f6883ee25e49caaea691ed2bc7339d] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp, totalSize=250.7 K 2023-04-09 09:56:13,887 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting ca62cc483bc247ae9bbbfb85e87a797d, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=164, earliestPutTs=1681034169170 2023-04-09 09:56:13,887 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting aed1071f46a94f9b82eacbf3a28f23fc, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1681034167895 2023-04-09 09:56:13,887 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 686c3ad2a7754a5e909b87deed1874ff, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1681034169230 2023-04-09 09:56:13,887 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting d3891a5d69534554b00b403ba504d73f, keycount=100, bloomtype=ROW, size=18.5 K, encoding=NONE, compression=NONE, seqNum=154, earliestPutTs=1681034168016 2023-04-09 09:56:13,888 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 65e557d0523d4babb81e58d469ef8ed5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=216, earliestPutTs=1681034169859 2023-04-09 09:56:13,890 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting d795c2cedb8943afaad609b09ea6dfcb, keycount=350, bloomtype=ROW, size=52.6 K, encoding=NONE, compression=NONE, seqNum=164, earliestPutTs=1681034169167 2023-04-09 09:56:13,891 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 74721bdf15624468891165c7f5afdb83, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=249, earliestPutTs=1681034170501 2023-04-09 09:56:13,891 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 033c11eb33fa4911a6ab53bc55dbac9d, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1681034169230 2023-04-09 09:56:13,891 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 2fd097b323724390ac79ca324ff52ac4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=261, earliestPutTs=1681034171127 2023-04-09 09:56:13,891 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 38eaff8221ad4f91a144403286f0080a, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=216, earliestPutTs=1681034169859 2023-04-09 09:56:13,891 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting a2925f0d1a1a4d188f830cd86c0d5ca4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=272, earliestPutTs=1681034173273 2023-04-09 09:56:13,891 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting e25930902a7346a5a1ac05688b0fc3ae, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=249, earliestPutTs=1681034170501 2023-04-09 09:56:13,892 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 22c905efc1c744049ad4d0d8cb342618, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=261, earliestPutTs=1681034171127 2023-04-09 09:56:13,892 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 19f6883ee25e49caaea691ed2bc7339d, keycount=350, bloomtype=ROW, size=52.9 K, encoding=NONE, compression=NONE, seqNum=272, earliestPutTs=1681034173267 2023-04-09 09:56:13,914 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): c8127616cf5355dd7f1db99764192b95#B#compaction#592 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:56:13,948 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] throttle.PressureAwareThroughputController(145): c8127616cf5355dd7f1db99764192b95#A#compaction#593 average throughput is 0.60 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:56:13,954 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/d462410689894f99a970feb1f8f68bb5 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/d462410689894f99a970feb1f8f68bb5 2023-04-09 09:56:13,957 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in c8127616cf5355dd7f1db99764192b95/B of c8127616cf5355dd7f1db99764192b95 into d462410689894f99a970feb1f8f68bb5(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:56:13,957 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c8127616cf5355dd7f1db99764192b95: 2023-04-09 09:56:13,957 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95., storeName=c8127616cf5355dd7f1db99764192b95/B, priority=10, startTime=1681034173884; duration=0sec 2023-04-09 09:56:13,957 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:56:13,957 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-09 09:56:13,959 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 84724 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-09 09:56:13,959 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): c8127616cf5355dd7f1db99764192b95/C is initiating minor compaction (all files) 2023-04-09 09:56:13,959 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c8127616cf5355dd7f1db99764192b95/C in TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:13,959 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/8e798f5fa9b5494a8e5e449294b29eae, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/297f6d7fced140c9ac8a7d2a705da177, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/e4312df7fdb04d74b6a119156a1e1685, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/926a1d5785144b4783b1224640ff7581, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/12fe1a6a4b79499ca7558b3e1cf5421a, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/00a3eb90e67d4fa9adb2e9841012c4e0, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/1d72fab54415439fbe236e8e25012ee6] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp, totalSize=82.7 K 2023-04-09 09:56:13,959 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 8e798f5fa9b5494a8e5e449294b29eae, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=154, earliestPutTs=1681034168006 2023-04-09 09:56:13,959 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 297f6d7fced140c9ac8a7d2a705da177, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=164, earliestPutTs=1681034169170 2023-04-09 09:56:13,959 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting e4312df7fdb04d74b6a119156a1e1685, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1681034169230 2023-04-09 09:56:13,960 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 926a1d5785144b4783b1224640ff7581, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=216, earliestPutTs=1681034169859 2023-04-09 09:56:13,960 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 12fe1a6a4b79499ca7558b3e1cf5421a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=249, earliestPutTs=1681034170501 2023-04-09 09:56:13,964 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 00a3eb90e67d4fa9adb2e9841012c4e0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=261, earliestPutTs=1681034171127 2023-04-09 09:56:13,965 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 1d72fab54415439fbe236e8e25012ee6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=272, earliestPutTs=1681034173273 2023-04-09 09:56:13,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on c8127616cf5355dd7f1db99764192b95 2023-04-09 09:56:13,975 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c8127616cf5355dd7f1db99764192b95 3/3 column families, dataSize=147.60 KB heapSize=415.45 KB 2023-04-09 09:56:13,976 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=A 2023-04-09 09:56:13,976 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:56:13,976 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=B 2023-04-09 09:56:13,976 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:56:13,976 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=C 2023-04-09 09:56:13,976 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:56:13,981 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:13,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:13,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:13,986 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): c8127616cf5355dd7f1db99764192b95#C#compaction#595 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:56:13,988 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/A/0b0b29af87db471da9925cf5270b588d as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/0b0b29af87db471da9925cf5270b588d 2023-04-09 09:56:13,991 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:13,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60506 deadline: 1681034233982, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:13,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:13,992 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:13,992 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:13,992 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:13,992 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:13,992 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:13,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60460 deadline: 1681034233983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,992 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:13,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60498 deadline: 1681034233988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,993 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:13,993 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:13,995 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:13,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:13,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:13,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:13,996 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in c8127616cf5355dd7f1db99764192b95/A of c8127616cf5355dd7f1db99764192b95 into 0b0b29af87db471da9925cf5270b588d(size=25.8 K), total size for store is 25.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:56:13,996 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c8127616cf5355dd7f1db99764192b95: 2023-04-09 09:56:13,996 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95., storeName=c8127616cf5355dd7f1db99764192b95/A, priority=8, startTime=1681034173884; duration=0sec 2023-04-09 09:56:13,996 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:56:13,996 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,996 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:13,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:13,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:13,997 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:13,997 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,997 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:13,998 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:13,998 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:13,998 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:13,998 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:13,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60436 deadline: 1681034233992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,998 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:13,999 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:13,999 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:13,999 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:13,999 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:13,999 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:13,999 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:13,999 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:13,999 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:13,999 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:13,999 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:13,999 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:13,999 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:14,001 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:14,001 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:14,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:14,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:14,002 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:14,002 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:14,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:14,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:14,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:14,003 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:14,003 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@174b53d9[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:14,003 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:14,003 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:14,003 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:14,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60448 deadline: 1681034233999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,004 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,004 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:14,004 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:14,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:14,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:14,004 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:14,004 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:14,005 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:14,006 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:14,006 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,006 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:14,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:14,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:14,006 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:14,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:14,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:14,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:14,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:14,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:14,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:14,010 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:14,010 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:14,010 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:14,010 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:14,010 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:14,010 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:14,015 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/C/cc0a064112ef4de5bba522b015032887 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/cc0a064112ef4de5bba522b015032887 2023-04-09 09:56:14,017 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:14,017 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:14,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:14,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:14,017 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:14,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:14,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:14,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:14,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:14,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:14,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:14,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:14,018 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:14,020 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:14,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:14,021 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304094ffc28f70a0542578b98b3146ae2602c to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304094ffc28f70a0542578b98b3146ae2602c 2023-04-09 09:56:14,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:14,022 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in c8127616cf5355dd7f1db99764192b95/C of c8127616cf5355dd7f1db99764192b95 into cc0a064112ef4de5bba522b015032887(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:56:14,022 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c8127616cf5355dd7f1db99764192b95: 2023-04-09 09:56:14,022 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95., storeName=c8127616cf5355dd7f1db99764192b95/C, priority=9, startTime=1681034173884; duration=0sec 2023-04-09 09:56:14,022 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:56:14,023 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:14,023 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:14,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:14,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:14,023 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:14,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:14,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:14,023 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,023 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:14,023 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:14,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:14,024 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:56:14,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:14,024 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:14,024 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,024 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:14,024 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:14,024 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:14,024 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:14,095 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:14,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60506 deadline: 1681034234092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,095 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:14,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60460 deadline: 1681034234093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,096 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:14,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60498 deadline: 1681034234094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,101 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:14,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60436 deadline: 1681034234100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,124 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:14,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:14,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:14,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:14,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:14,129 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:14,129 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:14,129 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:14,129 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:14,130 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:14,130 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:14,131 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:14,131 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:14,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:14,131 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,131 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:14,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:14,131 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:14,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:14,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:14,132 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:14,132 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:14,132 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:14,132 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:14,132 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:14,133 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:14,133 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:14,133 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:14,134 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:14,134 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:14,134 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:14,134 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:14,134 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:14,134 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:14,134 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,134 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:14,134 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:14,135 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,136 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:14,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:14,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:14,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:14,136 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:14,136 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:14,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:14,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:14,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:14,138 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:14,138 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d664d8a[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:14,138 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:14,138 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:14,139 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:14,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:14,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:14,139 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:14,139 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:14,139 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:14,139 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,139 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:14,141 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,141 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:14,141 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:14,141 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:14,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:14,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:14,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:14,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:14,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:14,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:14,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:14,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:14,144 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:14,144 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:14,144 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:14,145 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:14,145 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:14,145 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:14,146 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:14,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:14,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:14,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:14,146 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:14,146 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:14,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:14,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:14,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:14,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:14,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:14,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:14,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:14,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:14,148 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:14,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:14,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:14,149 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,151 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,151 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:14,151 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:14,151 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:14,151 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:14,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:14,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:14,151 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,151 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:14,151 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:14,151 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:14,151 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:14,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:14,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:14,152 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:14,152 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:14,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:14,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:14,252 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:14,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:14,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:14,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:14,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:14,256 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:14,256 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:14,256 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:14,256 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:14,257 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:14,257 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:14,258 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:14,258 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:14,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:14,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:14,259 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,259 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:14,259 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:14,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:14,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:14,260 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:14,260 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:14,260 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:14,260 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:14,260 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:14,260 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:14,260 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:14,260 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:14,260 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:14,260 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:14,261 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:14,261 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:14,261 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:14,261 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:14,261 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,261 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:14,261 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:14,262 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:14,262 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:14,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:14,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:14,263 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:14,263 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:14,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:14,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:14,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:14,265 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:14,265 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b772dd8[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:14,265 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:14,265 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:14,268 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:14,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:14,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:14,268 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:14,268 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:14,268 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:14,268 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,268 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:14,270 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:14,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:14,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:14,270 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:14,270 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:14,270 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:14,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:14,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:14,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:14,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:14,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:14,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:14,272 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:14,272 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:14,272 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:14,273 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:14,273 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:14,274 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:14,274 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:14,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:14,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:14,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:14,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:14,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:14,274 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:14,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:14,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:14,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:14,274 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:14,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:14,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:14,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:14,275 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:14,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:14,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,277 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,277 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:14,278 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:14,278 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:14,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:14,278 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:14,278 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:14,278 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:14,278 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:14,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:14,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:14,278 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:14,278 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:14,278 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:14,278 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:14,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:14,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:14,299 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:14,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60460 deadline: 1681034234296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,300 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:14,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60506 deadline: 1681034234297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,300 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:14,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60498 deadline: 1681034234298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,305 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:14,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60436 deadline: 1681034234303, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,378 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:14,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:14,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:14,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:14,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:14,382 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:14,382 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:14,382 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:14,382 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:14,382 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:14,382 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:14,386 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:14,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:14,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:14,386 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:14,387 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,387 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:14,387 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:14,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:14,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:14,387 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:14,388 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:14,388 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:14,388 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:14,388 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:14,388 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:14,388 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:14,388 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:14,388 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:14,388 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:14,388 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:14,388 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:14,388 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:14,388 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:14,388 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,388 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:14,388 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:14,390 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,390 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:14,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:14,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:14,390 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:14,390 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:14,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:14,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:14,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:14,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:14,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:14,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a5cb44d[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:14,392 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:14,392 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:14,393 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:14,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:14,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:14,393 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:14,393 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:14,393 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:14,393 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,393 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:14,394 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:14,394 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:14,394 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:14,394 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:14,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:14,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:14,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:14,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:14,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:14,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:14,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:14,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:14,397 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:14,397 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:14,397 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:14,397 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:14,397 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:14,399 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:14,399 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:14,399 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:14,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:14,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:14,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:14,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:14,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:14,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:14,400 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:14,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:14,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:14,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:14,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:14,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:14,401 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:14,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:14,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,406 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:14,406 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:14,406 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:14,406 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:56:14,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:14,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:14,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:14,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:14,406 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,406 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:14,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:14,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:14,406 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:14,406 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:14,406 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,406 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:14,406 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:14,406 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:14,407 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:14,426 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=310, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/A/160a7f9c11c24ccb820a59715df9d44b 2023-04-09 09:56:14,436 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=310 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/5dcb44ad7dc54c7aa2b4cf5c2ef0e4be 2023-04-09 09:56:14,445 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=310 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/C/017f5663b65f432eb9c959f973e9cbe0 2023-04-09 09:56:14,448 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/A/160a7f9c11c24ccb820a59715df9d44b as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/160a7f9c11c24ccb820a59715df9d44b 2023-04-09 09:56:14,451 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/160a7f9c11c24ccb820a59715df9d44b, entries=200, sequenceid=310, filesize=32.3 K 2023-04-09 09:56:14,452 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/5dcb44ad7dc54c7aa2b4cf5c2ef0e4be as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/5dcb44ad7dc54c7aa2b4cf5c2ef0e4be 2023-04-09 09:56:14,455 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/5dcb44ad7dc54c7aa2b4cf5c2ef0e4be, entries=150, sequenceid=310, filesize=11.9 K 2023-04-09 09:56:14,456 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/C/017f5663b65f432eb9c959f973e9cbe0 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/017f5663b65f432eb9c959f973e9cbe0 2023-04-09 09:56:14,459 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/017f5663b65f432eb9c959f973e9cbe0, entries=150, sequenceid=310, filesize=11.9 K 2023-04-09 09:56:14,459 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~415.41 KB/425376, currentSize=40.25 KB/41220 for c8127616cf5355dd7f1db99764192b95 in 484ms, sequenceid=310, compaction requested=false 2023-04-09 09:56:14,459 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c8127616cf5355dd7f1db99764192b95: 2023-04-09 09:56:14,507 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:14,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:14,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:14,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:14,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:14,510 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:14,510 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:14,510 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:14,510 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:14,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on c8127616cf5355dd7f1db99764192b95 2023-04-09 09:56:14,510 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c8127616cf5355dd7f1db99764192b95 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-09 09:56:14,511 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=A 2023-04-09 09:56:14,511 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:14,511 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:14,511 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:56:14,511 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=B 2023-04-09 09:56:14,511 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:56:14,511 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=C 2023-04-09 09:56:14,511 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:56:14,517 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:14,517 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:14,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:14,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:14,517 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,517 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:14,518 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:14,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:14,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:14,518 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:14,531 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:14,531 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:14,531 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:14,531 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:14,531 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:14,535 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:14,535 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:14,536 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:14,536 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:14,536 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:14,536 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:14,536 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:14,536 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:14,536 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,536 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:14,537 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:14,538 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,538 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:14,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:14,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:14,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:14,539 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:14,539 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:14,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:14,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:14,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:14,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:14,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@744e344[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:14,540 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:14,540 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:14,543 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:14,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:14,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:14,543 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:14,543 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:14,543 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:14,543 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,543 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:14,544 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:14,544 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:14,544 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:14,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:14,544 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:14,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:14,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:14,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:14,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:14,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:14,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:14,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:14,549 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:14,549 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:14,549 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:14,549 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:14,549 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:14,553 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:14,553 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:14,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:14,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:14,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:14,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:14,553 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:14,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:14,553 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040920b740b5f6bc47d58e7a4e7ba8860fb5 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040920b740b5f6bc47d58e7a4e7ba8860fb5 2023-04-09 09:56:14,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:14,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:14,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:14,554 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:14,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:14,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:14,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:14,555 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:14,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:14,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,557 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,557 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:14,557 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:14,557 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,557 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:14,557 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:14,557 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:14,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:14,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:14,557 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:14,557 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:14,557 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:14,558 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:14,558 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:14,558 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:14,558 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:14,558 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:56:14,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:14,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:14,572 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034174535, max timestamp=1681034174570Num uniques 859; 2023-04-09 09:56:14,572 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034174535, max timestamp=1681034174570Num uniques 859; 2023-04-09 09:56:14,572 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034174535, max timestamp=1681034174570Num uniques 859; 2023-04-09 09:56:14,572 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:56:14,572 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:56:14,572 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:56:14,613 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:14,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60506 deadline: 1681034234608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,613 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:14,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60436 deadline: 1681034234608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,614 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681034174595, max timestamp=1681034174612Num uniques 1018; 2023-04-09 09:56:14,614 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-09 09:56:14,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681034174595, max timestamp=1681034174612Num uniques 1018; 2023-04-09 09:56:14,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681034174595, max timestamp=1681034174612Num uniques 1018; 2023-04-09 09:56:14,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-09 09:56:14,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-09 09:56:14,636 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:14,636 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:14,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60460 deadline: 1681034234633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60498 deadline: 1681034234633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,638 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:14,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60448 deadline: 1681034234636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,658 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:14,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:14,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:14,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:14,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:14,666 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:14,666 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:14,667 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:14,667 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:14,667 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:14,667 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:14,668 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:14,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:14,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:14,668 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:14,668 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,668 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:14,669 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:14,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:14,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:14,669 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:14,669 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:14,669 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:14,669 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:14,669 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:14,669 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:14,675 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:14,675 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:14,680 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:14,680 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:14,680 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:14,680 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:14,680 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:14,680 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:14,680 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,680 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:14,680 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:14,682 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:14,682 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:14,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:14,682 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:14,682 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:14,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:14,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:14,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:14,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:14,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:14,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28e1f43b[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:14,685 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:14,685 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:14,686 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:14,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:14,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:14,686 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:14,686 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:14,686 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,686 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:14,686 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:14,688 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:14,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:14,688 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:14,689 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:14,689 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:14,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:14,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:14,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:14,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:14,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:14,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:14,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:14,692 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:14,692 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:14,692 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:14,692 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:14,692 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:14,694 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:14,694 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:14,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:14,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:14,694 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:14,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:14,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:14,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:14,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:14,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:14,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:14,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:14,695 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:14,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:14,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:14,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,696 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:14,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:14,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,700 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:14,700 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:14,700 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,700 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:14,700 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:14,700 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,700 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:14,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:14,700 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:56:14,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:14,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:14,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:14,700 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:14,700 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:14,700 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:14,701 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:14,701 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:14,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:14,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:14,715 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:14,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60506 deadline: 1681034234715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,742 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:14,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60460 deadline: 1681034234739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,743 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:14,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60448 deadline: 1681034234739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,743 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:14,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60498 deadline: 1681034234742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,801 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:14,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:14,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:14,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:14,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:14,805 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:14,805 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:14,805 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:14,805 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:14,805 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:14,806 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:14,807 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:14,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:14,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:14,807 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,807 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:14,807 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:14,808 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:14,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:14,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:14,808 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:14,808 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:14,809 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:14,809 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:14,809 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:14,809 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:14,812 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:14,812 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:14,812 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:14,812 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:14,812 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:14,812 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:14,812 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:14,812 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:14,813 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,813 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:14,813 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:14,815 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,815 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:14,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:14,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:14,816 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:14,816 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:14,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:14,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:14,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:14,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:14,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:14,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a8a1512[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:14,818 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:14,818 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:14,819 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:14,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:14,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:14,819 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:14,819 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:14,819 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:14,820 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,820 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:14,821 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:14,821 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:14,821 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:14,821 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:14,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:14,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:14,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:14,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:14,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:14,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:14,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:14,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:14,825 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:14,825 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:14,825 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:14,825 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:14,825 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:14,826 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:14,827 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:14,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:14,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:14,827 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:14,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:14,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:14,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:14,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:14,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:14,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:14,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:14,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:14,827 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:14,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:14,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,829 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:14,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:14,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,836 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,836 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:14,836 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:14,836 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:14,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:14,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:14,836 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:14,836 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,836 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:14,836 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:14,836 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:14,837 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:14,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:14,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:14,837 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:14,837 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:14,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:14,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:14,918 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:14,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60506 deadline: 1681034234916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,937 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:14,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:14,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:14,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:14,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:14,940 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:14,940 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:14,940 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:14,940 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:14,941 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:14,941 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:14,942 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:14,942 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:14,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:14,942 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,942 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:14,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:14,942 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:14,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:14,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:14,943 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:14,943 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:14,943 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:14,943 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:14,943 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:14,943 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:14,943 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:14,943 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:14,943 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:14,944 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:14,944 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:14,944 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:14,944 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:14,944 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:14,944 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,944 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:14,944 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:14,945 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:14,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60460 deadline: 1681034234943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,945 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:14,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60448 deadline: 1681034234944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,946 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,946 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:14,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:14,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:14,946 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:14,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60498 deadline: 1681034234945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:14,946 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:14,946 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:14,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:14,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:14,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:14,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:14,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@755b2cd2[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:14,948 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:14,948 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:14,950 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:14,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:14,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:14,950 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:14,950 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,950 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:14,950 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:14,950 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:14,951 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:14,951 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:14,951 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,951 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:14,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:14,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:14,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:14,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:14,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:14,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:14,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:14,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:14,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:14,954 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:14,954 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:14,954 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:14,954 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:14,954 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:14,955 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:14,955 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:14,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:14,955 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:14,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:14,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:14,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:14,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:14,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:14,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:14,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:14,956 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:14,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:14,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:14,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:14,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,957 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:14,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:14,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,959 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=322, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/A/3eb010383f974073a3f30742b41a0725 2023-04-09 09:56:14,960 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:14,960 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,960 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:14,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:14,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:14,960 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:14,960 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:14,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:14,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:14,960 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:14,960 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:14,960 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:14,960 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:14,960 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:14,960 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:14,961 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:14,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:14,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:14,969 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=322 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/08a64638841d437c9c1808dbcce13abb 2023-04-09 09:56:15,061 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:15,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:15,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:15,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:15,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:15,063 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:15,063 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:15,063 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:15,063 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:15,064 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:15,064 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:15,066 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:15,066 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:15,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:15,067 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,067 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:15,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:15,067 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:15,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:15,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:15,067 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:15,068 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:15,068 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:15,068 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:15,068 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:15,068 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:15,068 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:15,068 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:15,068 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:15,068 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:15,068 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:15,068 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:15,068 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:15,068 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:15,068 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,068 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:15,068 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:15,070 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,070 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:15,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:15,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:15,070 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:15,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:15,070 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:15,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:15,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:15,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:15,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:15,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5525affe[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:15,071 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:15,071 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:15,072 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:15,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:15,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:15,072 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:15,072 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:15,072 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:15,072 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,072 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:15,073 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,073 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:15,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:15,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:15,074 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:15,074 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:15,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:15,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:15,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:15,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:15,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:15,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:15,075 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:15,075 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:15,076 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:15,076 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:15,076 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:15,076 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:15,078 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:15,078 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:15,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:15,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:15,078 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:15,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:15,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:15,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:15,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:15,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:15,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:15,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:15,079 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:15,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:15,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:15,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,080 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:15,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:15,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,081 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,082 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:15,082 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:15,082 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:15,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:15,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:15,082 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:15,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:15,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:15,082 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:15,082 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,082 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:15,082 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:15,082 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:15,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:15,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:15,082 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:15,082 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:15,118 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:15,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60436 deadline: 1681034235116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,182 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:15,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:15,184 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:15,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:15,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:15,185 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:15,186 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:15,186 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:15,186 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:15,186 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:15,186 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:15,187 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:15,187 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:15,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:15,187 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,187 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:15,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:15,188 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:15,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:15,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:15,188 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:15,188 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:15,188 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:15,189 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:15,189 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:15,189 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:15,189 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:15,189 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:15,189 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:15,189 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:15,189 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:15,189 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:15,189 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:15,189 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:15,189 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,189 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:15,189 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:15,190 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,190 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:15,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:15,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:15,191 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:15,191 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:15,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:15,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:15,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:15,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:15,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:15,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72c2d94d[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:15,192 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:15,192 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:15,195 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:15,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:15,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:15,195 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,195 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:15,195 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:15,195 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:15,195 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:15,196 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:15,196 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:15,196 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,196 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:15,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:15,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:15,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:15,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:15,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:15,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:15,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:15,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:15,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:15,198 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:15,198 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:15,198 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:15,198 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:15,198 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:15,199 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:15,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:15,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:15,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:15,202 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:15,202 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:15,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:15,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:15,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:15,202 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:15,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:15,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:15,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:15,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:15,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:15,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,203 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:15,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:15,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,205 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:15,205 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,205 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:15,205 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:56:15,205 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:15,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:15,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:15,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:15,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:15,205 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:15,206 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:15,206 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:15,206 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,206 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:15,206 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:15,206 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:15,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:15,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:15,206 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:15,223 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:15,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60506 deadline: 1681034235221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,249 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:15,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60448 deadline: 1681034235248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,249 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:15,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60460 deadline: 1681034235248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,250 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:15,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60498 deadline: 1681034235248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,306 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:15,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:15,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:15,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:15,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:15,309 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:15,309 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:15,309 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:15,309 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:15,309 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:15,309 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:15,312 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,312 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:15,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:15,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:15,312 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,312 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:15,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:15,312 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:15,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:15,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:15,313 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:15,313 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:15,313 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:15,313 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:15,313 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:15,313 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:15,313 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:15,313 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:15,313 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:15,313 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:15,313 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:15,313 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:15,314 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:15,314 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:15,314 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,314 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:15,314 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:15,317 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,317 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:15,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:15,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:15,318 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:15,318 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:15,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:15,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:15,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:15,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:15,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:15,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@786a7082[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:15,319 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:15,319 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:15,320 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:15,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:15,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:15,320 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,320 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:15,320 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:15,320 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:15,320 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:15,321 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,321 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:15,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:15,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:15,321 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:15,321 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:15,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:15,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:15,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:15,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:15,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:15,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:15,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:15,323 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:15,323 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:15,323 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:15,323 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:15,323 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:15,325 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:15,325 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:15,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:15,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:15,325 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:15,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:15,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:15,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:15,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:15,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:15,325 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:15,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:15,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:15,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:15,326 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:15,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:15,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:15,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,329 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,330 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:15,330 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:15,330 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,330 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:15,330 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:15,330 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:15,330 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:15,330 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:15,330 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:15,330 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:15,330 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:15,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:15,330 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:15,330 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:56:15,330 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:15,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:15,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:15,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:15,376 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=322 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/C/01481e3b16bf439fbfc63e37b33733d0 2023-04-09 09:56:15,379 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/A/3eb010383f974073a3f30742b41a0725 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/3eb010383f974073a3f30742b41a0725 2023-04-09 09:56:15,384 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/3eb010383f974073a3f30742b41a0725, entries=200, sequenceid=322, filesize=32.3 K 2023-04-09 09:56:15,384 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/08a64638841d437c9c1808dbcce13abb as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/08a64638841d437c9c1808dbcce13abb 2023-04-09 09:56:15,387 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/08a64638841d437c9c1808dbcce13abb, entries=150, sequenceid=322, filesize=11.9 K 2023-04-09 09:56:15,388 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/C/01481e3b16bf439fbfc63e37b33733d0 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/01481e3b16bf439fbfc63e37b33733d0 2023-04-09 09:56:15,390 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/01481e3b16bf439fbfc63e37b33733d0, entries=150, sequenceid=322, filesize=11.9 K 2023-04-09 09:56:15,391 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for c8127616cf5355dd7f1db99764192b95 in 881ms, sequenceid=322, compaction requested=true 2023-04-09 09:56:15,391 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c8127616cf5355dd7f1db99764192b95: 2023-04-09 09:56:15,391 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:56:15,391 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:56:15,391 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-09 09:56:15,391 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-09 09:56:15,391 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:56:15,392 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 92628 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-09 09:56:15,392 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1912): c8127616cf5355dd7f1db99764192b95/A is initiating minor compaction (all files) 2023-04-09 09:56:15,392 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37002 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-09 09:56:15,392 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c8127616cf5355dd7f1db99764192b95/A in TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:15,392 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): c8127616cf5355dd7f1db99764192b95/B is initiating minor compaction (all files) 2023-04-09 09:56:15,392 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/0b0b29af87db471da9925cf5270b588d, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/160a7f9c11c24ccb820a59715df9d44b, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/3eb010383f974073a3f30742b41a0725] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp, totalSize=90.5 K 2023-04-09 09:56:15,392 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c8127616cf5355dd7f1db99764192b95/B in TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:15,392 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/d462410689894f99a970feb1f8f68bb5, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/5dcb44ad7dc54c7aa2b4cf5c2ef0e4be, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/08a64638841d437c9c1808dbcce13abb] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp, totalSize=36.1 K 2023-04-09 09:56:15,393 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 0b0b29af87db471da9925cf5270b588d, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=272, earliestPutTs=1681034173273 2023-04-09 09:56:15,393 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting d462410689894f99a970feb1f8f68bb5, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=272, earliestPutTs=1681034173273 2023-04-09 09:56:15,393 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 160a7f9c11c24ccb820a59715df9d44b, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=310, earliestPutTs=1681034173344 2023-04-09 09:56:15,393 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 5dcb44ad7dc54c7aa2b4cf5c2ef0e4be, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=310, earliestPutTs=1681034173344 2023-04-09 09:56:15,393 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 3eb010383f974073a3f30742b41a0725, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1681034173981 2023-04-09 09:56:15,393 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 08a64638841d437c9c1808dbcce13abb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1681034173981 2023-04-09 09:56:15,399 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): c8127616cf5355dd7f1db99764192b95#B#compaction#602 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:56:15,400 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] throttle.PressureAwareThroughputController(145): c8127616cf5355dd7f1db99764192b95#A#compaction#601 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:56:15,430 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:15,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:15,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:15,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:15,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:15,433 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:15,433 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:15,433 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:15,433 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:15,434 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:15,434 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:15,435 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:15,435 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:15,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:15,435 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,435 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:15,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:15,436 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:15,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:15,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:15,436 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:15,436 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:15,436 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:15,436 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:15,436 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:15,436 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:15,436 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:15,436 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:15,436 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:15,437 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:15,437 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c8127616cf5355dd7f1db99764192b95 3/3 column families, dataSize=134.18 KB heapSize=374.77 KB 2023-04-09 09:56:15,437 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=A 2023-04-09 09:56:15,437 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:56:15,437 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=B 2023-04-09 09:56:15,437 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:56:15,437 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=C 2023-04-09 09:56:15,437 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:56:15,453 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230409fee514a95677427bb58f5eb66d4063b1 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409fee514a95677427bb58f5eb66d4063b1 2023-04-09 09:56:15,461 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=374, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/A/4483b4a24bde47c19550c50aad380e57 2023-04-09 09:56:15,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on c8127616cf5355dd7f1db99764192b95 2023-04-09 09:56:15,727 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:15,751 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:15,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60506 deadline: 1681034235749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,753 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:15,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60460 deadline: 1681034235752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,753 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:15,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60498 deadline: 1681034235752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,754 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:15,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60448 deadline: 1681034235753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,812 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/9b5a960f42be4d21bb122a55040dce8e as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/9b5a960f42be4d21bb122a55040dce8e 2023-04-09 09:56:15,816 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c8127616cf5355dd7f1db99764192b95/B of c8127616cf5355dd7f1db99764192b95 into 9b5a960f42be4d21bb122a55040dce8e(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:56:15,816 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c8127616cf5355dd7f1db99764192b95: 2023-04-09 09:56:15,816 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95., storeName=c8127616cf5355dd7f1db99764192b95/B, priority=13, startTime=1681034175391; duration=0sec 2023-04-09 09:56:15,816 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:56:15,816 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-09 09:56:15,817 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36968 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-09 09:56:15,817 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): c8127616cf5355dd7f1db99764192b95/C is initiating minor compaction (all files) 2023-04-09 09:56:15,817 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c8127616cf5355dd7f1db99764192b95/C in TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:15,817 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/cc0a064112ef4de5bba522b015032887, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/017f5663b65f432eb9c959f973e9cbe0, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/01481e3b16bf439fbfc63e37b33733d0] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp, totalSize=36.1 K 2023-04-09 09:56:15,818 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting cc0a064112ef4de5bba522b015032887, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=272, earliestPutTs=1681034173273 2023-04-09 09:56:15,818 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 017f5663b65f432eb9c959f973e9cbe0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=310, earliestPutTs=1681034173344 2023-04-09 09:56:15,819 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 01481e3b16bf439fbfc63e37b33733d0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1681034173981 2023-04-09 09:56:15,820 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/A/a0eadfb2259b42c5a7a39558c75e0bed as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/a0eadfb2259b42c5a7a39558c75e0bed 2023-04-09 09:56:15,824 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): c8127616cf5355dd7f1db99764192b95#C#compaction#605 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:56:15,826 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c8127616cf5355dd7f1db99764192b95/A of c8127616cf5355dd7f1db99764192b95 into a0eadfb2259b42c5a7a39558c75e0bed(size=25.9 K), total size for store is 25.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:56:15,826 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c8127616cf5355dd7f1db99764192b95: 2023-04-09 09:56:15,826 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95., storeName=c8127616cf5355dd7f1db99764192b95/A, priority=13, startTime=1681034175391; duration=0sec 2023-04-09 09:56:15,826 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:56:15,855 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:15,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60506 deadline: 1681034235852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,872 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=374 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/7eb5e647f67b4eb4a7e350e687d51b43 2023-04-09 09:56:15,880 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=374 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/C/e7c5daf5a9c44f4e910cba69a4c97d55 2023-04-09 09:56:15,884 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/A/4483b4a24bde47c19550c50aad380e57 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/4483b4a24bde47c19550c50aad380e57 2023-04-09 09:56:15,888 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/4483b4a24bde47c19550c50aad380e57, entries=150, sequenceid=374, filesize=25.4 K 2023-04-09 09:56:15,889 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/7eb5e647f67b4eb4a7e350e687d51b43 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/7eb5e647f67b4eb4a7e350e687d51b43 2023-04-09 09:56:15,892 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/7eb5e647f67b4eb4a7e350e687d51b43, entries=150, sequenceid=374, filesize=11.9 K 2023-04-09 09:56:15,893 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/C/e7c5daf5a9c44f4e910cba69a4c97d55 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/e7c5daf5a9c44f4e910cba69a4c97d55 2023-04-09 09:56:15,896 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/e7c5daf5a9c44f4e910cba69a4c97d55, entries=150, sequenceid=374, filesize=11.9 K 2023-04-09 09:56:15,896 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~374.72 KB/383712, currentSize=46.96 KB/48090 for c8127616cf5355dd7f1db99764192b95 in 459ms, sequenceid=374, compaction requested=false 2023-04-09 09:56:15,896 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c8127616cf5355dd7f1db99764192b95: 2023-04-09 09:56:15,896 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:15,896 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:15,896 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:15,896 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,896 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:15,897 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:15,898 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,898 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:15,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:15,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:15,899 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:15,899 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:15,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:15,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:15,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:15,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:15,900 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:15,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@547beee1[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:15,901 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:15,901 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:15,902 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:15,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:15,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:15,902 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:15,902 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:15,902 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:15,902 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,902 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:15,904 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,904 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:15,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:15,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:15,905 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:15,905 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:15,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:15,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:15,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:15,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:15,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:15,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:15,907 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:15,907 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:15,907 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:15,907 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:15,907 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:15,907 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:15,909 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:15,909 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:15,909 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:15,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:15,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:15,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:15,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:15,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:15,909 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:15,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:15,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:15,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:15,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:15,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:15,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:15,910 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:15,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:15,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,912 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,913 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:15,913 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:15,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:15,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:15,913 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:15,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:15,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:15,913 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:56:15,913 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:15,913 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:15,913 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:15,913 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:15,913 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:15,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:15,913 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:15,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:15,913 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:15,913 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:16,013 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:16,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:16,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:16,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:16,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:16,017 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:16,017 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:16,017 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:16,017 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:16,017 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:16,017 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:16,018 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:16,018 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:16,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:16,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:16,018 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:16,018 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:16,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:16,019 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:16,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:16,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:16,019 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:16,019 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:16,020 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:16,020 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:16,020 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:16,020 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:16,020 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:16,020 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:16,020 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:16,020 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c8127616cf5355dd7f1db99764192b95 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-09 09:56:16,020 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=A 2023-04-09 09:56:16,021 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:56:16,021 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=B 2023-04-09 09:56:16,021 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:56:16,021 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=C 2023-04-09 09:56:16,021 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:56:16,034 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230409482c01af9b114e0eb62c1124b26a5e4c to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409482c01af9b114e0eb62c1124b26a5e4c 2023-04-09 09:56:16,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on c8127616cf5355dd7f1db99764192b95 2023-04-09 09:56:16,062 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:16,112 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034176098, max timestamp=1681034176110Num uniques 859; 2023-04-09 09:56:16,112 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:56:16,112 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034176098, max timestamp=1681034176110Num uniques 859; 2023-04-09 09:56:16,112 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034176098, max timestamp=1681034176110Num uniques 859; 2023-04-09 09:56:16,112 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:56:16,112 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:56:16,145 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681034176131, max timestamp=1681034176143Num uniques 1018; 2023-04-09 09:56:16,145 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-09 09:56:16,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681034176131, max timestamp=1681034176143Num uniques 1018; 2023-04-09 09:56:16,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-09 09:56:16,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681034176131, max timestamp=1681034176143Num uniques 1018; 2023-04-09 09:56:16,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-09 09:56:16,173 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:16,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60436 deadline: 1681034236169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:16,176 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:16,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60506 deadline: 1681034236173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:16,240 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/C/b1f06794470843268a4a21f19fb38ff5 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/b1f06794470843268a4a21f19fb38ff5 2023-04-09 09:56:16,244 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c8127616cf5355dd7f1db99764192b95/C of c8127616cf5355dd7f1db99764192b95 into b1f06794470843268a4a21f19fb38ff5(size=12.4 K), total size for store is 24.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:56:16,244 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c8127616cf5355dd7f1db99764192b95: 2023-04-09 09:56:16,244 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95., storeName=c8127616cf5355dd7f1db99764192b95/C, priority=13, startTime=1681034175391; duration=0sec 2023-04-09 09:56:16,244 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:56:16,280 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:16,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60506 deadline: 1681034236279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:16,280 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:16,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60436 deadline: 1681034236279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:16,444 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=386, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/A/6ae9fd5c7b9f442e977e8c77a2eba500 2023-04-09 09:56:16,458 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=386 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/1bc7772601404601af8caa10342d6c16 2023-04-09 09:56:16,483 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:16,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60436 deadline: 1681034236481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:16,484 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:16,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60506 deadline: 1681034236482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:16,757 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:16,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60498 deadline: 1681034236754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:16,761 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:16,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60460 deadline: 1681034236759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:16,761 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:16,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60448 deadline: 1681034236760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:16,785 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:16,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60436 deadline: 1681034236784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:16,785 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:16,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60506 deadline: 1681034236785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:16,882 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=386 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/C/7aa789a8fabf4cc586e926b1d7e75856 2023-04-09 09:56:16,886 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/A/6ae9fd5c7b9f442e977e8c77a2eba500 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/6ae9fd5c7b9f442e977e8c77a2eba500 2023-04-09 09:56:16,889 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/6ae9fd5c7b9f442e977e8c77a2eba500, entries=100, sequenceid=386, filesize=18.6 K 2023-04-09 09:56:16,891 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/1bc7772601404601af8caa10342d6c16 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/1bc7772601404601af8caa10342d6c16 2023-04-09 09:56:16,895 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/1bc7772601404601af8caa10342d6c16, entries=100, sequenceid=386, filesize=9.5 K 2023-04-09 09:56:16,895 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/C/7aa789a8fabf4cc586e926b1d7e75856 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/7aa789a8fabf4cc586e926b1d7e75856 2023-04-09 09:56:16,900 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/7aa789a8fabf4cc586e926b1d7e75856, entries=100, sequenceid=386, filesize=9.5 K 2023-04-09 09:56:16,901 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for c8127616cf5355dd7f1db99764192b95 in 881ms, sequenceid=386, compaction requested=true 2023-04-09 09:56:16,901 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c8127616cf5355dd7f1db99764192b95: 2023-04-09 09:56:16,901 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:16,901 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:16,901 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:16,901 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:16,901 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:16,901 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:16,903 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:16,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:16,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:16,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:16,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:16,903 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:16,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:16,903 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:16,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:16,904 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:16,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:16,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:16,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:16,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2cd9f1cf[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:16,904 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:16,904 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:16,906 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:16,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:16,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:16,906 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:16,906 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:16,906 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:16,907 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:16,906 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:16,908 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:16,908 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:16,908 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:16,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:16,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:16,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:16,908 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:16,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:16,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:16,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:16,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:16,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:16,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:16,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:16,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:16,910 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:16,911 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:16,911 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:16,911 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:16,911 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:16,911 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:16,912 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:16,912 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:16,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:16,912 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:16,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:16,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:16,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:16,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:16,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:16,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:16,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:16,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:16,913 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:16,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:16,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:16,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:16,914 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:16,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:16,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:16,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:16,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:16,917 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:16,917 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:16,917 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:16,917 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:16,917 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:16,917 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:16,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:16,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:16,917 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:16,917 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:16,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:16,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:16,918 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:16,918 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:16,917 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:16,918 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:56:16,917 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:16,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:16,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:17,018 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:17,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:17,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:17,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:17,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:17,021 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:17,021 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:17,021 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:17,021 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:17,022 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:17,022 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:17,023 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:17,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:17,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:17,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:17,024 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:17,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:17,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:17,024 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:17,024 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,027 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,027 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:17,028 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:17,028 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:17,028 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:17,028 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:17,028 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:17,028 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:17,028 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:17,028 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:17,028 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:17,029 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c8127616cf5355dd7f1db99764192b95 3/3 column families, dataSize=134.18 KB heapSize=379.45 KB 2023-04-09 09:56:17,029 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=A 2023-04-09 09:56:17,029 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:56:17,029 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=B 2023-04-09 09:56:17,029 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:56:17,029 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=C 2023-04-09 09:56:17,029 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:56:17,048 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040991db2eb18c5f4ae3ac0b48527f5a0171 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040991db2eb18c5f4ae3ac0b48527f5a0171 2023-04-09 09:56:17,058 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=440, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/A/16840ed44c3848278ab30b389bdb74a9 2023-04-09 09:56:17,076 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=440 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/8891fd4138e64238a0fe3194fdda1d5d 2023-04-09 09:56:17,094 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=440 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/C/f8d5e50acaeb4d8b9379202ce1469830 2023-04-09 09:56:17,100 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/A/16840ed44c3848278ab30b389bdb74a9 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/16840ed44c3848278ab30b389bdb74a9 2023-04-09 09:56:17,104 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/16840ed44c3848278ab30b389bdb74a9, entries=150, sequenceid=440, filesize=25.4 K 2023-04-09 09:56:17,105 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/8891fd4138e64238a0fe3194fdda1d5d as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/8891fd4138e64238a0fe3194fdda1d5d 2023-04-09 09:56:17,108 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/8891fd4138e64238a0fe3194fdda1d5d, entries=150, sequenceid=440, filesize=11.9 K 2023-04-09 09:56:17,109 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/C/f8d5e50acaeb4d8b9379202ce1469830 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/f8d5e50acaeb4d8b9379202ce1469830 2023-04-09 09:56:17,112 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/f8d5e50acaeb4d8b9379202ce1469830, entries=150, sequenceid=440, filesize=11.9 K 2023-04-09 09:56:17,113 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~379.41 KB/388512, currentSize=0 B/0 for c8127616cf5355dd7f1db99764192b95 in 84ms, sequenceid=440, compaction requested=true 2023-04-09 09:56:17,113 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c8127616cf5355dd7f1db99764192b95: 2023-04-09 09:56:17,113 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:17,113 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:17,113 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:17,113 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,113 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:17,113 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:17,115 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,115 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:17,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:17,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:17,115 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:17,116 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:17,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:17,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:17,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:17,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:17,117 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:17,117 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48b54da8[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:17,117 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:17,117 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:17,121 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:17,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:17,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:17,121 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:17,121 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:17,121 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:17,121 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,121 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:17,122 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:17,122 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,122 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:17,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:17,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:17,122 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:17,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:17,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:17,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:17,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:17,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:17,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:17,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:17,125 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:17,125 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:17,125 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:17,125 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:17,125 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:17,127 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:17,127 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:17,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:17,127 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:17,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:17,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:17,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:17,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:17,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:17,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:17,128 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:17,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:17,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:17,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:17,137 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:17,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:17,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:17,140 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,140 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:17,140 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:17,140 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,140 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:17,140 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:17,140 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:17,141 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:17,141 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:17,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:17,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:17,141 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:17,142 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:17,141 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:17,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:17,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:17,144 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:56:17,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:17,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:17,242 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:17,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:17,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:17,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:17,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:17,246 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:17,246 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:17,246 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:17,246 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:17,247 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:17,247 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:17,248 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:17,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:17,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:17,248 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:17,248 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,249 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:17,250 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:17,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:17,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:17,250 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:17,250 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:17,250 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:17,251 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:17,251 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:17,251 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:17,251 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:17,251 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:17,251 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:17,251 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:17,251 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c8127616cf5355dd7f1db99764192b95: 2023-04-09 09:56:17,251 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:17,251 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:17,251 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:17,252 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,252 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:17,252 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:17,253 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:17,253 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:17,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:17,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:17,254 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:17,254 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:17,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:17,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:17,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:17,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:17,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63f0ca5c[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:17,255 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:17,255 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:17,256 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:17,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:17,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:17,256 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:17,256 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:17,256 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:17,256 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,257 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:17,258 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:17,258 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,258 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:17,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:17,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:17,258 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:17,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:17,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:17,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:17,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:17,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:17,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:17,260 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:17,260 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:17,260 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:17,260 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:17,260 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:17,260 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:17,263 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:17,263 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:17,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:17,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:17,263 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:17,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:17,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:17,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:17,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:17,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:17,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:17,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:17,265 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:17,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:17,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:17,266 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:17,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:17,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,269 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:17,269 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,269 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:17,269 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:17,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:17,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:17,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:17,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:17,270 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:17,270 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:17,269 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:17,270 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:17,270 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,270 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:17,270 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:17,270 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:17,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:17,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:17,271 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:56:17,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on c8127616cf5355dd7f1db99764192b95 2023-04-09 09:56:17,322 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c8127616cf5355dd7f1db99764192b95 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-09 09:56:17,322 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=A 2023-04-09 09:56:17,322 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:56:17,323 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=B 2023-04-09 09:56:17,323 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:56:17,323 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=C 2023-04-09 09:56:17,323 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:56:17,341 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230409ba0475327b90422b8adce979a114a25b to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409ba0475327b90422b8adce979a114a25b 2023-04-09 09:56:17,352 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=452, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/A/a2be7e31374d4492bbb2217370ff0db0 2023-04-09 09:56:17,361 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=452 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/bed95e971b124685834f0015e59cf56a 2023-04-09 09:56:17,370 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:17,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:17,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:17,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:17,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:17,373 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:17,373 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:17,373 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:17,373 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:17,374 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:17,374 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:17,375 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:17,375 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:17,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:17,376 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,376 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:17,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:17,377 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:17,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:17,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:17,377 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:17,377 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:17,377 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:17,377 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:17,377 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:17,378 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:17,378 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:17,378 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:17,378 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:17,378 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:17,378 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:17,378 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:17,378 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:17,379 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:17,379 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,379 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:17,379 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:17,380 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:17,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:17,380 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:17,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:17,381 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:17,381 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:17,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:17,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:17,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:17,383 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034177371, max timestamp=1681034177381Num uniques 859; 2023-04-09 09:56:17,383 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:56:17,383 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034177371, max timestamp=1681034177381Num uniques 859; 2023-04-09 09:56:17,383 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034177371, max timestamp=1681034177381Num uniques 859; 2023-04-09 09:56:17,383 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:56:17,383 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:56:17,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:17,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68f73c25[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:17,383 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:17,384 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:17,385 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:17,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:17,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:17,385 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:17,385 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:17,385 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:17,385 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,386 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:17,391 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,391 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:17,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:17,391 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:17,391 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:17,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:17,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:17,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:17,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:17,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:17,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:17,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:17,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:17,395 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:17,395 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:17,395 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:17,395 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:17,395 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:17,398 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:17,398 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:17,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:17,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:17,398 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:17,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:17,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:17,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:17,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:17,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:17,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:17,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:17,399 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:17,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:17,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:17,400 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:17,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:17,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,402 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,403 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:17,403 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:17,403 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:17,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:17,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:17,403 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:17,403 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,403 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:17,403 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:17,403 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:17,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:17,403 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:17,403 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:56:17,403 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:17,403 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:17,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:17,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:17,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:17,420 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:17,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60436 deadline: 1681034237415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,424 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:17,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60506 deadline: 1681034237420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,503 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:17,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:17,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:17,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:17,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:17,508 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:17,508 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:17,508 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:17,508 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:17,509 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:17,509 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:17,510 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:17,511 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:17,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:17,511 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,511 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:17,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:17,511 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:17,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:17,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:17,512 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:17,512 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:17,512 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:17,512 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:17,512 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:17,512 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:17,515 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:17,515 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:17,516 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:17,516 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:17,516 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:17,516 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:17,516 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:17,516 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:17,516 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,516 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:17,516 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:17,517 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,517 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:17,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:17,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:17,518 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:17,518 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:17,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:17,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:17,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:17,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:17,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:17,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b43ff7d[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:17,519 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:17,519 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:17,520 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:17,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:17,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:17,521 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:17,521 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:17,521 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:17,521 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,521 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:17,522 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,522 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:17,522 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:17,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:17,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:17,522 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:17,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:17,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:17,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:17,523 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:17,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60436 deadline: 1681034237521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:17,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:17,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:17,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:17,525 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:17,525 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:17,525 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:17,525 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:17,525 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:17,528 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:17,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60506 deadline: 1681034237525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,530 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:17,530 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:17,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:17,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:17,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:17,530 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:17,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:17,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:17,530 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:17,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:17,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:17,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:17,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:17,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:17,532 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:17,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:17,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:17,534 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:17,534 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:17,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:17,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:17,534 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:17,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:17,534 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:17,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:17,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:17,534 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:17,535 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:17,535 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,535 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:17,535 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:17,535 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:56:17,535 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:17,535 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:17,535 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:17,635 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:17,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:17,637 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:17,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:17,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:17,639 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:17,639 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:17,639 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:17,639 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:17,639 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:17,639 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:17,640 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,640 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:17,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:17,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:17,641 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,641 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:17,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:17,641 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:17,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:17,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:17,641 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:17,641 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:17,642 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:17,642 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:17,642 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:17,642 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:17,642 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:17,642 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:17,642 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:17,642 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:17,642 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:17,642 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:17,642 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:17,643 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:17,643 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,643 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:17,643 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:17,644 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,645 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:17,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:17,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:17,645 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:17,645 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:17,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:17,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:17,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:17,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:17,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:17,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69792bb0[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:17,647 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:17,647 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:17,648 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:17,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:17,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:17,648 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:17,648 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:17,648 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:17,648 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,648 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:17,649 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,649 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:17,649 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:17,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:17,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:17,649 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:17,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:17,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:17,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:17,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:17,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:17,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:17,652 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:17,652 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:17,652 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:17,652 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:17,652 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:17,652 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:17,659 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:17,659 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:17,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:17,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:17,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:17,659 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:17,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:17,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:17,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:17,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:17,660 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:17,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:17,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:17,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:17,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:17,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,661 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:17,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:17,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,665 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:17,665 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:17,665 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:17,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:17,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:17,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:17,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:17,665 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,665 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:17,665 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:17,665 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,665 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:17,665 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:17,665 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:17,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:17,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:17,666 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:17,666 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:17,727 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:17,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60436 deadline: 1681034237724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,732 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:17,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60506 deadline: 1681034237729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,766 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:17,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:17,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:17,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:17,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:17,777 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:17,777 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:17,777 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:17,777 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:17,777 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:17,778 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:17,779 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:17,779 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:17,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:17,780 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,780 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:17,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:17,780 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:17,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:17,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:17,780 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:17,781 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:17,781 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:17,781 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:17,781 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:17,781 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:17,781 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=452 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/C/c193eb6a766e4e6fb90cdf47a6f9a81a 2023-04-09 09:56:17,784 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:17,784 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:17,784 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:17,784 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:17,784 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:17,785 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:17,785 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:17,785 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:17,785 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,785 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:17,785 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:17,785 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/A/a2be7e31374d4492bbb2217370ff0db0 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/a2be7e31374d4492bbb2217370ff0db0 2023-04-09 09:56:17,786 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:17,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:17,786 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:17,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:17,787 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:17,787 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:17,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:17,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:17,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:17,789 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/a2be7e31374d4492bbb2217370ff0db0, entries=250, sequenceid=452, filesize=39.2 K 2023-04-09 09:56:17,789 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:17,789 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@cbebee9[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:17,789 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:17,789 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:17,789 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/bed95e971b124685834f0015e59cf56a as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/bed95e971b124685834f0015e59cf56a 2023-04-09 09:56:17,790 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:17,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:17,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:17,790 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:17,790 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:17,790 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:17,791 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,791 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:17,792 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,792 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:17,792 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:17,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:17,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:17,792 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:17,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:17,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:17,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:17,793 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/bed95e971b124685834f0015e59cf56a, entries=150, sequenceid=452, filesize=11.9 K 2023-04-09 09:56:17,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:17,794 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/C/c193eb6a766e4e6fb90cdf47a6f9a81a as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/c193eb6a766e4e6fb90cdf47a6f9a81a 2023-04-09 09:56:17,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:17,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:17,795 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:17,795 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:17,795 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:17,795 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:17,795 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:17,795 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:17,797 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/c193eb6a766e4e6fb90cdf47a6f9a81a, entries=150, sequenceid=452, filesize=11.9 K 2023-04-09 09:56:17,798 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for c8127616cf5355dd7f1db99764192b95 in 476ms, sequenceid=452, compaction requested=true 2023-04-09 09:56:17,798 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c8127616cf5355dd7f1db99764192b95: 2023-04-09 09:56:17,798 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:17,798 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:17,798 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:56:17,798 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-09 09:56:17,798 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-09 09:56:17,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:17,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:17,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:17,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:17,798 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,798 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:56:17,798 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:17,799 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:56:17,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:17,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:17,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:17,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:17,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:17,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:17,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:17,801 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:17,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:17,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:17,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,803 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,803 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:17,803 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:17,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:17,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:17,803 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:17,803 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:17,803 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:17,803 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:56:17,803 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,803 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:17,803 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:17,803 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:17,804 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:17,804 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:17,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:17,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:17,804 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 137778 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-09 09:56:17,804 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1912): c8127616cf5355dd7f1db99764192b95/A is initiating minor compaction (all files) 2023-04-09 09:56:17,804 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c8127616cf5355dd7f1db99764192b95/A in TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:17,804 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/a0eadfb2259b42c5a7a39558c75e0bed, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/4483b4a24bde47c19550c50aad380e57, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/6ae9fd5c7b9f442e977e8c77a2eba500, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/16840ed44c3848278ab30b389bdb74a9, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/a2be7e31374d4492bbb2217370ff0db0] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp, totalSize=134.5 K 2023-04-09 09:56:17,805 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59032 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-09 09:56:17,805 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): c8127616cf5355dd7f1db99764192b95/B is initiating minor compaction (all files) 2023-04-09 09:56:17,805 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c8127616cf5355dd7f1db99764192b95/B in TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:17,805 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/9b5a960f42be4d21bb122a55040dce8e, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/7eb5e647f67b4eb4a7e350e687d51b43, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/1bc7772601404601af8caa10342d6c16, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/8891fd4138e64238a0fe3194fdda1d5d, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/bed95e971b124685834f0015e59cf56a] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp, totalSize=57.6 K 2023-04-09 09:56:17,805 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 9b5a960f42be4d21bb122a55040dce8e, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1681034173981 2023-04-09 09:56:17,806 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 7eb5e647f67b4eb4a7e350e687d51b43, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1681034174631 2023-04-09 09:56:17,806 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting a0eadfb2259b42c5a7a39558c75e0bed, keycount=150, bloomtype=ROW, size=25.9 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1681034173981 2023-04-09 09:56:17,807 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 1bc7772601404601af8caa10342d6c16, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=386, earliestPutTs=1681034175745 2023-04-09 09:56:17,807 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 4483b4a24bde47c19550c50aad380e57, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1681034174631 2023-04-09 09:56:17,807 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 8891fd4138e64238a0fe3194fdda1d5d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=440, earliestPutTs=1681034176160 2023-04-09 09:56:17,808 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting bed95e971b124685834f0015e59cf56a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=452, earliestPutTs=1681034177299 2023-04-09 09:56:17,808 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 6ae9fd5c7b9f442e977e8c77a2eba500, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=386, earliestPutTs=1681034175745 2023-04-09 09:56:17,808 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 16840ed44c3848278ab30b389bdb74a9, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=440, earliestPutTs=1681034176160 2023-04-09 09:56:17,809 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting a2be7e31374d4492bbb2217370ff0db0, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=452, earliestPutTs=1681034177299 2023-04-09 09:56:17,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:17,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:17,821 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): c8127616cf5355dd7f1db99764192b95#B#compaction#616 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:56:17,826 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] throttle.PressureAwareThroughputController(145): c8127616cf5355dd7f1db99764192b95#A#compaction#617 average throughput is 2.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:56:17,903 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:17,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:17,906 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:17,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:17,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:17,907 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:17,907 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:17,907 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:17,907 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:17,908 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:17,908 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:17,909 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:17,909 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:17,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:17,910 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:17,910 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:17,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:17,912 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:17,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:17,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:17,913 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:17,914 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:17,914 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:17,914 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:17,914 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:17,916 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:17,916 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:17,916 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:17,916 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:17,916 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c8127616cf5355dd7f1db99764192b95 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-09 09:56:17,917 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=A 2023-04-09 09:56:17,918 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:56:17,918 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=B 2023-04-09 09:56:17,919 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:56:17,919 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=C 2023-04-09 09:56:17,919 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:56:17,947 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304091f9833f5a670455893c11217db309c75 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304091f9833f5a670455893c11217db309c75 2023-04-09 09:56:17,965 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=488, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/A/89af7261fb1d4960909a8ceabec39ae9 2023-04-09 09:56:18,015 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=488 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/4e4d3cb1eb734e0ab002aded175a0808 2023-04-09 09:56:18,038 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=488 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/C/385a4d80b7b345a8a875d4d7e2c95e4a 2023-04-09 09:56:18,042 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/A/89af7261fb1d4960909a8ceabec39ae9 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/89af7261fb1d4960909a8ceabec39ae9 2023-04-09 09:56:18,046 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/89af7261fb1d4960909a8ceabec39ae9, entries=150, sequenceid=488, filesize=25.4 K 2023-04-09 09:56:18,048 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/4e4d3cb1eb734e0ab002aded175a0808 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/4e4d3cb1eb734e0ab002aded175a0808 2023-04-09 09:56:18,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on c8127616cf5355dd7f1db99764192b95 2023-04-09 09:56:18,051 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:18,055 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/4e4d3cb1eb734e0ab002aded175a0808, entries=150, sequenceid=488, filesize=11.9 K 2023-04-09 09:56:18,056 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/C/385a4d80b7b345a8a875d4d7e2c95e4a as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/385a4d80b7b345a8a875d4d7e2c95e4a 2023-04-09 09:56:18,069 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/385a4d80b7b345a8a875d4d7e2c95e4a, entries=150, sequenceid=488, filesize=11.9 K 2023-04-09 09:56:18,075 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=13.42 KB/13740 for c8127616cf5355dd7f1db99764192b95 in 159ms, sequenceid=488, compaction requested=true 2023-04-09 09:56:18,076 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c8127616cf5355dd7f1db99764192b95: 2023-04-09 09:56:18,076 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:18,076 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:18,076 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:18,076 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,076 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:18,076 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:18,077 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,077 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:18,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:18,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:18,078 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:18,078 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:18,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:18,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:18,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:18,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:18,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:18,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5df9ad4f[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:18,080 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:18,080 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:18,082 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:18,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:18,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:18,082 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:18,082 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:18,082 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:18,082 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,082 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:18,083 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,083 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:18,083 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:18,083 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:18,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:18,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:18,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:18,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:18,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:18,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:18,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:18,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:18,087 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:18,087 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:18,087 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:18,087 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:18,087 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:18,087 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:18,088 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:18,088 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:18,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:18,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:18,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:18,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:18,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:18,089 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:18,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:18,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:18,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:18,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:18,090 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:18,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:18,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:18,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,091 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:18,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:18,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,097 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,097 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:18,097 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:18,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:18,097 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:56:18,097 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:18,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:18,097 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,097 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:18,097 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:18,097 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:18,097 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:18,097 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:18,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:18,098 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:18,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:18,098 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:18,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:18,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:18,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on c8127616cf5355dd7f1db99764192b95 2023-04-09 09:56:18,102 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c8127616cf5355dd7f1db99764192b95 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-09 09:56:18,102 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=A 2023-04-09 09:56:18,102 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:56:18,102 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=B 2023-04-09 09:56:18,102 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:56:18,102 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=C 2023-04-09 09:56:18,102 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:56:18,128 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230409bcb8a376a560429190e6da5118abb952 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409bcb8a376a560429190e6da5118abb952 2023-04-09 09:56:18,144 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=498, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/A/9fd43c0a453d40518e12bdb9585d4bb6 2023-04-09 09:56:18,153 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681034178132, max timestamp=1681034178151Num uniques 859; 2023-04-09 09:56:18,154 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681034178132, max timestamp=1681034178151Num uniques 859; 2023-04-09 09:56:18,154 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681034178132, max timestamp=1681034178151Num uniques 859; 2023-04-09 09:56:18,154 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-09 09:56:18,154 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-09 09:56:18,154 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-09 09:56:18,158 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=498 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/52928e7397b3411882eb72c8766e5a08 2023-04-09 09:56:18,198 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:18,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:18,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:18,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:18,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:18,201 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:18,201 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:18,201 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:18,201 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:18,202 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:18,202 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:18,203 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:18,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:18,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:18,203 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:18,203 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,203 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:18,203 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:18,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:18,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:18,204 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:18,204 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:18,204 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:18,204 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:18,204 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:18,204 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:18,204 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:18,205 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:18,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60506 deadline: 1681034238199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,205 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:18,205 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:18,205 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:18,205 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:18,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60436 deadline: 1681034238199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,205 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:18,205 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:18,205 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:18,205 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:18,205 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,205 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:18,205 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:18,207 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:18,207 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:18,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:18,207 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:18,207 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:18,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:18,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:18,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:18,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:18,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:18,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4955199a[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:18,210 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:18,210 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:18,211 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:18,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:18,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:18,211 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:18,211 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:18,211 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:18,213 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:18,213 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:18,213 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,213 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:18,214 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:18,214 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:18,214 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:18,214 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:18,214 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:18,214 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:18,214 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:18,216 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:18,216 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:18,216 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:18,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:18,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:18,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:18,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:18,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:18,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:18,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:18,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:18,216 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:18,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:18,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:18,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:18,218 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:18,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:18,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,220 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,220 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:18,220 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:18,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:18,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:18,220 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:18,220 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:18,220 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:18,220 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:18,220 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,220 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:18,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:18,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:18,220 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:18,220 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:18,220 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:18,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:18,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:18,257 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/e181c99d2b344c8db6dc3e88c6e97ed0 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/e181c99d2b344c8db6dc3e88c6e97ed0 2023-04-09 09:56:18,261 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in c8127616cf5355dd7f1db99764192b95/B of c8127616cf5355dd7f1db99764192b95 into e181c99d2b344c8db6dc3e88c6e97ed0(size=12.6 K), total size for store is 24.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:56:18,262 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c8127616cf5355dd7f1db99764192b95: 2023-04-09 09:56:18,262 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95., storeName=c8127616cf5355dd7f1db99764192b95/B, priority=11, startTime=1681034177798; duration=0sec 2023-04-09 09:56:18,262 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:56:18,262 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-09 09:56:18,264 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71184 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-09 09:56:18,264 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): c8127616cf5355dd7f1db99764192b95/C is initiating minor compaction (all files) 2023-04-09 09:56:18,264 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c8127616cf5355dd7f1db99764192b95/C in TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:18,264 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/b1f06794470843268a4a21f19fb38ff5, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/e7c5daf5a9c44f4e910cba69a4c97d55, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/7aa789a8fabf4cc586e926b1d7e75856, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/f8d5e50acaeb4d8b9379202ce1469830, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/c193eb6a766e4e6fb90cdf47a6f9a81a, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/385a4d80b7b345a8a875d4d7e2c95e4a] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp, totalSize=69.5 K 2023-04-09 09:56:18,265 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting b1f06794470843268a4a21f19fb38ff5, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1681034173981 2023-04-09 09:56:18,265 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting e7c5daf5a9c44f4e910cba69a4c97d55, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1681034174631 2023-04-09 09:56:18,265 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 7aa789a8fabf4cc586e926b1d7e75856, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=386, earliestPutTs=1681034175745 2023-04-09 09:56:18,266 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting f8d5e50acaeb4d8b9379202ce1469830, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=440, earliestPutTs=1681034176160 2023-04-09 09:56:18,266 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting c193eb6a766e4e6fb90cdf47a6f9a81a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=452, earliestPutTs=1681034177299 2023-04-09 09:56:18,266 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 385a4d80b7b345a8a875d4d7e2c95e4a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=488, earliestPutTs=1681034177409 2023-04-09 09:56:18,277 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): c8127616cf5355dd7f1db99764192b95#C#compaction#624 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:56:18,292 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/A/2009135c156044ce8c3d4a3633067e56 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/2009135c156044ce8c3d4a3633067e56 2023-04-09 09:56:18,297 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in c8127616cf5355dd7f1db99764192b95/A of c8127616cf5355dd7f1db99764192b95 into 2009135c156044ce8c3d4a3633067e56(size=26.1 K), total size for store is 51.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:56:18,297 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c8127616cf5355dd7f1db99764192b95: 2023-04-09 09:56:18,297 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95., storeName=c8127616cf5355dd7f1db99764192b95/A, priority=11, startTime=1681034177798; duration=0sec 2023-04-09 09:56:18,297 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:56:18,309 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:18,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60506 deadline: 1681034238306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,309 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:18,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60436 deadline: 1681034238306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,320 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:18,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:18,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:18,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:18,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:18,324 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:18,324 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:18,324 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:18,324 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:18,325 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:18,325 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:18,326 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:18,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:18,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:18,326 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:18,327 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,327 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:18,327 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:18,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:18,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:18,327 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:18,327 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:18,327 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:18,328 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:18,328 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:18,328 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:18,336 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:18,336 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:18,336 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:18,336 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:18,336 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:18,336 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:18,336 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:18,336 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:18,336 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,336 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:18,336 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:18,338 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,338 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:18,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:18,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:18,338 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:18,338 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:18,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:18,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:18,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:18,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:18,340 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:18,340 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69518fab[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:18,340 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:18,340 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:18,341 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:18,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:18,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:18,341 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:18,341 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:18,341 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:18,342 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,342 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:18,343 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:18,343 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:18,343 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:18,343 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:18,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:18,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:18,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:18,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:18,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:18,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:18,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:18,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:18,348 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:18,349 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:18,349 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:18,349 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:18,349 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:18,350 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:18,350 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:18,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:18,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:18,350 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:18,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:18,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:18,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:18,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:18,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:18,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:18,351 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:18,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:18,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:18,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:18,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:18,353 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:18,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,355 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,356 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:18,356 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:18,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:18,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:18,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:18,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:18,356 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:18,356 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,356 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:18,356 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:18,356 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:18,356 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:18,356 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:18,356 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:18,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:18,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:18,356 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:18,456 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:18,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:18,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:18,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:18,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:18,464 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:18,464 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:18,464 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:18,464 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:18,465 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:18,465 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:18,466 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:18,466 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,466 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,466 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:18,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:18,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:18,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:18,468 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:18,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:18,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:18,469 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:18,469 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:18,469 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:18,469 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:18,469 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:18,469 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:18,472 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:18,472 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:18,472 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:18,472 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:18,472 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:18,472 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:18,472 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:18,472 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:18,472 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,472 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:18,472 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:18,474 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,474 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:18,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:18,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:18,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:18,474 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:18,474 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:18,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:18,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:18,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:18,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:18,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10612d9e[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:18,480 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:18,480 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:18,483 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:18,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:18,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:18,484 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,484 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:18,486 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:18,486 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:18,486 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:18,487 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,488 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:18,488 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:18,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:18,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:18,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:18,488 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:18,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:18,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:18,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:18,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:18,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:18,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:18,496 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:18,496 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:18,496 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:18,496 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:18,496 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:18,499 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:18,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:18,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:18,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:18,499 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:18,499 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:18,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:18,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:18,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:18,499 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:18,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:18,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:18,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:18,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:18,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:18,501 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:18,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:18,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,506 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,506 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:18,506 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:18,506 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:18,507 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,507 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:18,507 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:18,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:18,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:18,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:18,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:18,507 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:18,507 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:18,507 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:18,507 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:18,507 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:18,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:18,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:18,512 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:18,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60506 deadline: 1681034238510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,514 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:18,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60436 deadline: 1681034238511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,567 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=498 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/C/885c6865791d4036a35b56549bb81901 2023-04-09 09:56:18,571 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/A/9fd43c0a453d40518e12bdb9585d4bb6 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/9fd43c0a453d40518e12bdb9585d4bb6 2023-04-09 09:56:18,575 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/9fd43c0a453d40518e12bdb9585d4bb6, entries=250, sequenceid=498, filesize=39.2 K 2023-04-09 09:56:18,575 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/52928e7397b3411882eb72c8766e5a08 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/52928e7397b3411882eb72c8766e5a08 2023-04-09 09:56:18,581 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/52928e7397b3411882eb72c8766e5a08, entries=150, sequenceid=498, filesize=11.9 K 2023-04-09 09:56:18,582 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/C/885c6865791d4036a35b56549bb81901 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/885c6865791d4036a35b56549bb81901 2023-04-09 09:56:18,586 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/885c6865791d4036a35b56549bb81901, entries=150, sequenceid=498, filesize=11.9 K 2023-04-09 09:56:18,587 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for c8127616cf5355dd7f1db99764192b95 in 485ms, sequenceid=498, compaction requested=true 2023-04-09 09:56:18,587 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c8127616cf5355dd7f1db99764192b95: 2023-04-09 09:56:18,587 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:56:18,587 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:56:18,587 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-09 09:56:18,587 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-09 09:56:18,588 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 92904 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-09 09:56:18,588 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1912): c8127616cf5355dd7f1db99764192b95/A is initiating minor compaction (all files) 2023-04-09 09:56:18,588 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c8127616cf5355dd7f1db99764192b95/A in TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:18,588 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/2009135c156044ce8c3d4a3633067e56, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/89af7261fb1d4960909a8ceabec39ae9, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/9fd43c0a453d40518e12bdb9585d4bb6] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp, totalSize=90.7 K 2023-04-09 09:56:18,595 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 2009135c156044ce8c3d4a3633067e56, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=452, earliestPutTs=1681034177299 2023-04-09 09:56:18,595 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 89af7261fb1d4960909a8ceabec39ae9, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=488, earliestPutTs=1681034177409 2023-04-09 09:56:18,596 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 9fd43c0a453d40518e12bdb9585d4bb6, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=498, earliestPutTs=1681034178084 2023-04-09 09:56:18,607 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:18,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:18,611 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:18,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:18,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:18,612 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:18,612 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:18,612 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:18,612 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:18,613 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:18,613 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:18,613 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] throttle.PressureAwareThroughputController(145): c8127616cf5355dd7f1db99764192b95#A#compaction#625 average throughput is 2.19 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:56:18,614 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:18,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:18,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:18,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:18,614 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,614 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:18,615 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,615 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:18,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:18,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:18,615 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:18,615 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:18,615 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:18,615 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:18,615 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:18,615 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:18,615 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:18,616 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:18,616 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:18,616 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:18,616 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c8127616cf5355dd7f1db99764192b95 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-09 09:56:18,616 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=A 2023-04-09 09:56:18,616 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:56:18,616 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=B 2023-04-09 09:56:18,616 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:56:18,616 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=C 2023-04-09 09:56:18,616 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:56:18,673 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230409b3647897a3a740cba2903a6914b9e35e to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409b3647897a3a740cba2903a6914b9e35e 2023-04-09 09:56:18,688 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=536, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/A/df548ec2fb014b64b8969818d4680fde 2023-04-09 09:56:18,709 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/C/658303b5f67e46c98ae1ecc98ca89592 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/658303b5f67e46c98ae1ecc98ca89592 2023-04-09 09:56:18,713 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=536 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/021b5da8dca54f2fb004b648849a503a 2023-04-09 09:56:18,714 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in c8127616cf5355dd7f1db99764192b95/C of c8127616cf5355dd7f1db99764192b95 into 658303b5f67e46c98ae1ecc98ca89592(size=12.6 K), total size for store is 24.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:56:18,714 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c8127616cf5355dd7f1db99764192b95: 2023-04-09 09:56:18,714 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95., storeName=c8127616cf5355dd7f1db99764192b95/C, priority=10, startTime=1681034177799; duration=0sec 2023-04-09 09:56:18,715 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-09 09:56:18,715 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-09 09:56:18,716 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-09 09:56:18,716 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): c8127616cf5355dd7f1db99764192b95/B is initiating minor compaction (all files) 2023-04-09 09:56:18,716 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c8127616cf5355dd7f1db99764192b95/B in TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:18,716 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/e181c99d2b344c8db6dc3e88c6e97ed0, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/4e4d3cb1eb734e0ab002aded175a0808, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/52928e7397b3411882eb72c8766e5a08] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp, totalSize=36.4 K 2023-04-09 09:56:18,717 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting e181c99d2b344c8db6dc3e88c6e97ed0, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=452, earliestPutTs=1681034177299 2023-04-09 09:56:18,718 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 4e4d3cb1eb734e0ab002aded175a0808, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=488, earliestPutTs=1681034177409 2023-04-09 09:56:18,720 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 52928e7397b3411882eb72c8766e5a08, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=498, earliestPutTs=1681034178088 2023-04-09 09:56:18,729 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): c8127616cf5355dd7f1db99764192b95#B#compaction#629 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:56:18,737 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=536 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/C/93f897978079415082b8d01880747086 2023-04-09 09:56:18,742 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/A/df548ec2fb014b64b8969818d4680fde as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/df548ec2fb014b64b8969818d4680fde 2023-04-09 09:56:18,745 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/24ccfbb87828407eb69fdc2c91aef52f as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/24ccfbb87828407eb69fdc2c91aef52f 2023-04-09 09:56:18,748 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/df548ec2fb014b64b8969818d4680fde, entries=150, sequenceid=536, filesize=25.4 K 2023-04-09 09:56:18,749 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/021b5da8dca54f2fb004b648849a503a as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/021b5da8dca54f2fb004b648849a503a 2023-04-09 09:56:18,750 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c8127616cf5355dd7f1db99764192b95/B of c8127616cf5355dd7f1db99764192b95 into 24ccfbb87828407eb69fdc2c91aef52f(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:56:18,750 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c8127616cf5355dd7f1db99764192b95: 2023-04-09 09:56:18,750 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95., storeName=c8127616cf5355dd7f1db99764192b95/B, priority=13, startTime=1681034178587; duration=0sec 2023-04-09 09:56:18,750 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:56:18,750 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-09 09:56:18,751 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-09 09:56:18,751 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-09 09:56:18,751 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. because compaction request was cancelled 2023-04-09 09:56:18,754 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/021b5da8dca54f2fb004b648849a503a, entries=150, sequenceid=536, filesize=11.9 K 2023-04-09 09:56:18,754 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/C/93f897978079415082b8d01880747086 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/93f897978079415082b8d01880747086 2023-04-09 09:56:18,758 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/93f897978079415082b8d01880747086, entries=150, sequenceid=536, filesize=11.9 K 2023-04-09 09:56:18,759 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=0 B/0 for c8127616cf5355dd7f1db99764192b95 in 143ms, sequenceid=536, compaction requested=true 2023-04-09 09:56:18,759 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c8127616cf5355dd7f1db99764192b95: 2023-04-09 09:56:18,760 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:18,760 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:18,760 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:18,760 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,760 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:18,760 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:18,762 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:18,762 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:18,762 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:18,763 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:18,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:18,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:18,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:18,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:18,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:18,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:18,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e27908[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:18,765 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:18,765 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:18,767 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:18,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:18,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:18,768 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:18,768 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:18,768 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,768 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:18,768 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:18,769 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:18,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:18,770 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:18,770 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:18,770 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:18,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:18,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:18,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:18,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:18,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:18,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:18,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:18,773 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:18,773 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:18,773 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:18,773 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:18,773 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:18,775 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:18,775 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:18,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:18,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:18,775 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:18,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:18,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:18,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:18,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:18,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:18,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:18,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:18,777 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:18,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:18,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:18,778 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:18,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:18,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,780 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,780 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:18,781 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:18,781 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,781 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:18,781 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:18,781 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:18,781 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:18,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:18,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:18,781 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:18,781 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:18,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:18,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:18,781 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:18,781 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:18,781 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:56:18,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:18,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:18,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on c8127616cf5355dd7f1db99764192b95 2023-04-09 09:56:18,808 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c8127616cf5355dd7f1db99764192b95 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-09 09:56:18,809 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=A 2023-04-09 09:56:18,809 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:56:18,809 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=B 2023-04-09 09:56:18,809 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:56:18,809 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=C 2023-04-09 09:56:18,809 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:56:18,841 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:18,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60460 deadline: 1681034238835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,841 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:18,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60436 deadline: 1681034238836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,841 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:18,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60498 deadline: 1681034238836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,842 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034178825, max timestamp=1681034178839Num uniques 859; 2023-04-09 09:56:18,842 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:56:18,842 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034178825, max timestamp=1681034178840Num uniques 859; 2023-04-09 09:56:18,842 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034178825, max timestamp=1681034178840Num uniques 859; 2023-04-09 09:56:18,842 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:56:18,842 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:56:18,881 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:18,881 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:18,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60448 deadline: 1681034238879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:18,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:18,885 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:18,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:18,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:18,898 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:18,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60506 deadline: 1681034238882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,898 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:18,898 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:18,898 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:18,899 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:18,899 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:18,901 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:18,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:18,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:18,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:18,902 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:18,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:18,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:18,903 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:18,906 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,907 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,907 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:18,908 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:18,908 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:18,909 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:18,909 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:18,909 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:18,909 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:18,909 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:18,909 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:18,909 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:18,909 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:18,909 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:18,909 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:18,909 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:18,909 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,909 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:18,910 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:18,911 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:18,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:18,911 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:18,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:18,912 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:18,912 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:18,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:18,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:18,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:18,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:18,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@611a3947[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:18,913 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:18,913 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:18,914 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:18,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:18,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:18,914 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:18,914 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:18,914 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:18,915 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,916 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:18,916 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:18,916 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:18,916 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,916 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:18,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:18,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:18,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:18,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:18,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:18,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:18,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:18,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:18,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:18,919 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:18,919 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:18,919 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:18,919 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:18,919 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:18,922 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:18,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:18,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:18,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:18,922 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:18,923 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:18,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:18,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:18,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:18,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:18,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:18,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:18,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:18,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:18,924 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:18,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:18,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:18,930 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,932 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,932 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:18,932 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:18,932 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,932 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:18,932 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:18,932 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:18,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:18,933 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:18,933 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:18,932 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:18,932 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:18,933 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:18,933 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:18,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:18,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:18,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:18,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:18,945 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:18,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60460 deadline: 1681034238942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,946 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:18,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60498 deadline: 1681034238942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,946 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:18,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60436 deadline: 1681034238944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:18,985 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:18,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60448 deadline: 1681034238983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,000 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:19,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60506 deadline: 1681034238999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,033 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:19,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:19,035 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:19,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:19,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:19,036 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:19,036 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:19,036 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:19,036 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:19,036 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:19,037 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:19,038 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,038 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:19,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:19,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:19,038 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,038 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:19,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:19,038 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:19,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:19,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:19,039 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:19,039 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:19,039 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:19,039 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:19,039 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:19,039 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:19,039 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:19,039 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:19,039 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:19,039 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:19,039 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:19,039 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:19,039 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:19,039 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:19,039 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,040 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:19,040 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:19,041 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,041 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:19,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:19,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:19,041 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:19,041 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:19,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:19,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:19,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:19,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:19,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:19,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@453de18[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:19,042 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:19,042 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:19,044 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:19,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:19,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:19,044 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:19,044 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:19,044 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,044 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:19,044 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:19,046 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,046 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:19,046 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:19,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:19,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:19,046 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:19,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:19,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:19,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:19,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:19,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:19,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:19,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:19,048 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:19,048 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:19,048 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:19,048 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:19,048 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:19,049 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:19,049 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:19,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:19,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:19,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:19,049 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:19,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:19,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:19,050 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:19,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:19,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:19,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:19,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:19,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:19,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:19,051 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:19,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:19,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,053 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,053 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:19,053 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:19,053 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:56:19,053 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:19,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:19,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:19,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:19,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:19,053 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:19,053 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:19,053 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:19,053 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:19,053 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,054 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:19,054 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:19,054 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:19,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:19,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:19,061 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/A/facafe1ddc5c4257abae997a789723a4 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/facafe1ddc5c4257abae997a789723a4 2023-04-09 09:56:19,065 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c8127616cf5355dd7f1db99764192b95/A of c8127616cf5355dd7f1db99764192b95 into facafe1ddc5c4257abae997a789723a4(size=26.2 K), total size for store is 51.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:56:19,065 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c8127616cf5355dd7f1db99764192b95: 2023-04-09 09:56:19,065 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95., storeName=c8127616cf5355dd7f1db99764192b95/A, priority=13, startTime=1681034178587; duration=0sec 2023-04-09 09:56:19,065 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:56:19,150 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:19,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60436 deadline: 1681034239147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,150 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:19,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60460 deadline: 1681034239147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,150 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:19,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60498 deadline: 1681034239148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,153 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:19,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:19,156 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:19,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:19,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:19,157 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:19,157 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:19,158 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:19,158 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:19,158 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:19,158 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:19,165 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:19,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:19,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:19,165 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:19,166 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:19,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:19,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:19,166 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:19,166 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,166 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:19,167 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:19,167 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:19,167 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:19,167 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:19,167 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:19,167 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:19,167 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:19,167 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:19,167 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:19,168 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:19,168 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:19,168 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:19,168 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:19,168 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,168 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:19,168 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:19,172 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,172 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:19,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:19,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:19,172 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:19,172 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:19,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:19,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:19,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:19,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:19,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:19,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1dc71ed1[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:19,178 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:19,178 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:19,180 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:19,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:19,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:19,180 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:19,180 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:19,180 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:19,180 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,180 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:19,182 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,182 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:19,182 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:19,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:19,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:19,182 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:19,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:19,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:19,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:19,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:19,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:19,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:19,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:19,185 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:19,185 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:19,185 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:19,185 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:19,185 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:19,186 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:19,186 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:19,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:19,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:19,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:19,186 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:19,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:19,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:19,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:19,187 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:19,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:19,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:19,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:19,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:19,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:19,188 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:19,188 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60448 deadline: 1681034239188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:19,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:19,190 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:19,190 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:19,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:19,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:19,190 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:19,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:19,191 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:56:19,191 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:19,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:19,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:19,190 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:19,191 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:19,191 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:19,191 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:19,191 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,191 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:19,191 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:19,191 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:19,206 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:19,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60506 deadline: 1681034239202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,223 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230409f0e73684a70f415bbc2ee3037e632477 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409f0e73684a70f415bbc2ee3037e632477 2023-04-09 09:56:19,232 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=548, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/A/7642c2b4cefe404585320ea97682e5df 2023-04-09 09:56:19,262 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=548 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/21a9451b38c449a685d6662b6bce364b 2023-04-09 09:56:19,279 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=548 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/C/9a844e5165f14aca9ada2f473349aaf0 2023-04-09 09:56:19,283 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/A/7642c2b4cefe404585320ea97682e5df as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/7642c2b4cefe404585320ea97682e5df 2023-04-09 09:56:19,286 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/7642c2b4cefe404585320ea97682e5df, entries=300, sequenceid=548, filesize=46.1 K 2023-04-09 09:56:19,287 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/21a9451b38c449a685d6662b6bce364b as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/21a9451b38c449a685d6662b6bce364b 2023-04-09 09:56:19,289 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/21a9451b38c449a685d6662b6bce364b, entries=150, sequenceid=548, filesize=11.9 K 2023-04-09 09:56:19,290 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/C/9a844e5165f14aca9ada2f473349aaf0 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/9a844e5165f14aca9ada2f473349aaf0 2023-04-09 09:56:19,291 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:19,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:19,293 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/9a844e5165f14aca9ada2f473349aaf0, entries=150, sequenceid=548, filesize=11.9 K 2023-04-09 09:56:19,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:19,293 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for c8127616cf5355dd7f1db99764192b95 in 485ms, sequenceid=548, compaction requested=true 2023-04-09 09:56:19,294 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c8127616cf5355dd7f1db99764192b95: 2023-04-09 09:56:19,294 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:56:19,294 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-09 09:56:19,294 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-09 09:56:19,294 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:56:19,294 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:56:19,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:19,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:19,294 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:19,295 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:19,295 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 100046 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-09 09:56:19,295 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:19,295 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:19,295 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1912): c8127616cf5355dd7f1db99764192b95/A is initiating minor compaction (all files) 2023-04-09 09:56:19,295 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37376 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-09 09:56:19,295 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c8127616cf5355dd7f1db99764192b95/A in TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:19,295 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): c8127616cf5355dd7f1db99764192b95/B is initiating minor compaction (all files) 2023-04-09 09:56:19,295 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/facafe1ddc5c4257abae997a789723a4, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/df548ec2fb014b64b8969818d4680fde, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/7642c2b4cefe404585320ea97682e5df] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp, totalSize=97.7 K 2023-04-09 09:56:19,295 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c8127616cf5355dd7f1db99764192b95/B in TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:19,295 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/24ccfbb87828407eb69fdc2c91aef52f, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/021b5da8dca54f2fb004b648849a503a, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/21a9451b38c449a685d6662b6bce364b] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp, totalSize=36.5 K 2023-04-09 09:56:19,296 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:19,296 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 24ccfbb87828407eb69fdc2c91aef52f, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=498, earliestPutTs=1681034178088 2023-04-09 09:56:19,296 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting facafe1ddc5c4257abae997a789723a4, keycount=150, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=498, earliestPutTs=1681034178088 2023-04-09 09:56:19,296 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:19,296 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 021b5da8dca54f2fb004b648849a503a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=536, earliestPutTs=1681034178184 2023-04-09 09:56:19,296 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting df548ec2fb014b64b8969818d4680fde, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=536, earliestPutTs=1681034178184 2023-04-09 09:56:19,296 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 21a9451b38c449a685d6662b6bce364b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=548, earliestPutTs=1681034178797 2023-04-09 09:56:19,296 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 7642c2b4cefe404585320ea97682e5df, keycount=300, bloomtype=ROW, size=46.1 K, encoding=NONE, compression=NONE, seqNum=548, earliestPutTs=1681034178797 2023-04-09 09:56:19,297 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:19,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:19,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:19,297 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:19,297 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,297 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:19,298 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:19,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:19,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:19,298 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:19,299 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:19,299 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:19,299 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:19,299 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:19,299 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:19,300 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:19,300 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:19,300 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:19,301 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:19,301 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c8127616cf5355dd7f1db99764192b95 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-09 09:56:19,301 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=A 2023-04-09 09:56:19,301 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:56:19,301 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=B 2023-04-09 09:56:19,301 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:56:19,301 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=C 2023-04-09 09:56:19,301 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:56:19,302 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): c8127616cf5355dd7f1db99764192b95#B#compaction#633 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:56:19,313 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] throttle.PressureAwareThroughputController(145): c8127616cf5355dd7f1db99764192b95#A#compaction#634 average throughput is 1.97 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:56:19,325 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230409cdf0bfadc03346b8a70e971265f354dc to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409cdf0bfadc03346b8a70e971265f354dc 2023-04-09 09:56:19,333 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=586, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/A/b127ecd481d54190b3f45c859710248a 2023-04-09 09:56:19,339 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/A/f2527b9174e74bf99667a59e5f75d544 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/f2527b9174e74bf99667a59e5f75d544 2023-04-09 09:56:19,343 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c8127616cf5355dd7f1db99764192b95/A of c8127616cf5355dd7f1db99764192b95 into f2527b9174e74bf99667a59e5f75d544(size=26.3 K), total size for store is 26.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:56:19,343 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c8127616cf5355dd7f1db99764192b95: 2023-04-09 09:56:19,343 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95., storeName=c8127616cf5355dd7f1db99764192b95/A, priority=13, startTime=1681034179294; duration=0sec 2023-04-09 09:56:19,343 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:56:19,343 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-09 09:56:19,344 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49460 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-09 09:56:19,345 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1912): c8127616cf5355dd7f1db99764192b95/C is initiating minor compaction (all files) 2023-04-09 09:56:19,345 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c8127616cf5355dd7f1db99764192b95/C in TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:19,345 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/658303b5f67e46c98ae1ecc98ca89592, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/885c6865791d4036a35b56549bb81901, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/93f897978079415082b8d01880747086, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/9a844e5165f14aca9ada2f473349aaf0] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp, totalSize=48.3 K 2023-04-09 09:56:19,345 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 658303b5f67e46c98ae1ecc98ca89592, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=488, earliestPutTs=1681034177409 2023-04-09 09:56:19,345 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=586 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/b86f3364105b47fdad86f9c1e2616c99 2023-04-09 09:56:19,346 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 885c6865791d4036a35b56549bb81901, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=498, earliestPutTs=1681034178088 2023-04-09 09:56:19,346 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 93f897978079415082b8d01880747086, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=536, earliestPutTs=1681034178184 2023-04-09 09:56:19,346 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 9a844e5165f14aca9ada2f473349aaf0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=548, earliestPutTs=1681034178797 2023-04-09 09:56:19,353 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] throttle.PressureAwareThroughputController(145): c8127616cf5355dd7f1db99764192b95#C#compaction#638 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:56:19,354 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=586 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/C/365ea5ede1e94dada6f1176d0c6190f3 2023-04-09 09:56:19,367 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/A/b127ecd481d54190b3f45c859710248a as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/b127ecd481d54190b3f45c859710248a 2023-04-09 09:56:19,369 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/C/4a0d633a6ca14844925fd937da30fc1e as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/4a0d633a6ca14844925fd937da30fc1e 2023-04-09 09:56:19,371 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/b127ecd481d54190b3f45c859710248a, entries=150, sequenceid=586, filesize=25.4 K 2023-04-09 09:56:19,371 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/b86f3364105b47fdad86f9c1e2616c99 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/b86f3364105b47fdad86f9c1e2616c99 2023-04-09 09:56:19,374 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c8127616cf5355dd7f1db99764192b95/C of c8127616cf5355dd7f1db99764192b95 into 4a0d633a6ca14844925fd937da30fc1e(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:56:19,374 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c8127616cf5355dd7f1db99764192b95: 2023-04-09 09:56:19,374 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95., storeName=c8127616cf5355dd7f1db99764192b95/C, priority=12, startTime=1681034179294; duration=0sec 2023-04-09 09:56:19,374 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:56:19,375 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/b86f3364105b47fdad86f9c1e2616c99, entries=150, sequenceid=586, filesize=11.9 K 2023-04-09 09:56:19,375 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/C/365ea5ede1e94dada6f1176d0c6190f3 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/365ea5ede1e94dada6f1176d0c6190f3 2023-04-09 09:56:19,381 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/365ea5ede1e94dada6f1176d0c6190f3, entries=150, sequenceid=586, filesize=11.9 K 2023-04-09 09:56:19,382 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=0 B/0 for c8127616cf5355dd7f1db99764192b95 in 81ms, sequenceid=586, compaction requested=false 2023-04-09 09:56:19,382 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c8127616cf5355dd7f1db99764192b95: 2023-04-09 09:56:19,382 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:19,382 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:19,382 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:19,382 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,382 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:19,382 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:19,383 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:19,383 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:19,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:19,384 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:19,384 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:19,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:19,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:19,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:19,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:19,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:19,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e7e0e58[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:19,385 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:19,385 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:19,386 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:19,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:19,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:19,387 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:19,387 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:19,387 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:19,387 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,387 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:19,394 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,394 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:19,394 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:19,394 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:19,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:19,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:19,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:19,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:19,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:19,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:19,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:19,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:19,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:19,397 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:19,397 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:19,397 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:19,397 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:19,397 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:19,398 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:19,398 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:19,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:19,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:19,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:19,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:19,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:19,398 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:19,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:19,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:19,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:19,399 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:19,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:19,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:19,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:19,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,400 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:19,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:19,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,402 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:19,402 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,402 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:19,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:19,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:19,402 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:19,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:19,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:19,402 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:19,403 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:19,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:19,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:19,403 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,403 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:19,403 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:19,403 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:19,403 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:19,403 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:19,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on c8127616cf5355dd7f1db99764192b95 2023-04-09 09:56:19,489 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c8127616cf5355dd7f1db99764192b95 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-09 09:56:19,490 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=A 2023-04-09 09:56:19,490 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:56:19,490 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=B 2023-04-09 09:56:19,490 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:56:19,490 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=C 2023-04-09 09:56:19,490 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:56:19,503 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:19,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:19,505 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230409401b6bf5f79a4ef687039407ed3efcbc to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409401b6bf5f79a4ef687039407ed3efcbc 2023-04-09 09:56:19,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:19,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:19,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:19,508 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:19,508 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:19,508 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:19,508 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:19,508 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:19,508 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:19,510 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,510 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:19,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:19,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:19,510 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,510 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:19,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:19,510 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:19,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:19,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:19,511 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:19,520 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:19,520 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:19,520 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:19,520 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:19,520 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:19,525 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:19,525 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:19,525 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:19,525 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:19,525 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:19,525 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:19,525 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:19,525 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:19,525 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,526 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:19,526 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:19,528 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:19,528 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:19,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:19,529 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:19,529 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:19,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:19,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:19,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:19,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:19,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:19,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e9eced1[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:19,531 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:19,531 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:19,532 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:19,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:19,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:19,532 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:19,532 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:19,532 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:19,532 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,532 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:19,533 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,533 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:19,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:19,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:19,534 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:19,534 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:19,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:19,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:19,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:19,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:19,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:19,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:19,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:19,537 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:19,537 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:19,537 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:19,537 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:19,537 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:19,538 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:19,538 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:19,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:19,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:19,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:19,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:19,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:19,538 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:19,539 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:19,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:19,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:19,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:19,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:19,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:19,541 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034179534, max timestamp=1681034179539Num uniques 859; 2023-04-09 09:56:19,541 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:56:19,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:19,541 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034179534, max timestamp=1681034179539Num uniques 859; 2023-04-09 09:56:19,541 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:56:19,542 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:19,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:19,543 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034179534, max timestamp=1681034179539Num uniques 859; 2023-04-09 09:56:19,543 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:56:19,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,545 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,545 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:19,545 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:19,545 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:19,545 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:19,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:19,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:19,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:19,545 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:19,545 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,545 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:19,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:19,545 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:19,546 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:19,545 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:19,546 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:19,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:19,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:19,558 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:19,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60448 deadline: 1681034239553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,559 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:19,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60436 deadline: 1681034239554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,559 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:19,559 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:19,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60498 deadline: 1681034239554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60506 deadline: 1681034239555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,559 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:19,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60460 deadline: 1681034239557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,646 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:19,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:19,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:19,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:19,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:19,652 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:19,652 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:19,652 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:19,652 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:19,652 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:19,653 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:19,654 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,654 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:19,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:19,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:19,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:19,654 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,655 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:19,655 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:19,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:19,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:19,655 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:19,662 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:19,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60448 deadline: 1681034239660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,662 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:19,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60436 deadline: 1681034239660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,662 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:19,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60460 deadline: 1681034239660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,663 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:19,664 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:19,664 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:19,664 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:19,664 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:19,664 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:19,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60506 deadline: 1681034239660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,666 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:19,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60498 deadline: 1681034239664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,669 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:19,669 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:19,669 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:19,669 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:19,669 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:19,669 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:19,669 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:19,669 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:19,669 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,669 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:19,669 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:19,671 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:19,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:19,671 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:19,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:19,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:19,674 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:19,674 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:19,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:19,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:19,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:19,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24276404[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:19,675 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:19,676 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:19,680 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:19,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:19,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:19,681 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:19,681 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:19,681 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:19,681 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,681 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:19,683 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,683 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:19,683 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:19,683 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:19,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:19,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:19,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:19,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:19,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:19,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:19,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:19,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:19,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:19,689 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:19,690 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:19,690 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:19,690 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:19,690 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:19,693 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:19,693 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:19,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:19,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:19,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:19,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:19,693 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:19,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:19,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:19,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:19,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:19,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:19,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:19,694 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:19,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:19,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:19,695 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:19,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,697 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:19,697 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:19,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:19,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:19,698 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,698 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:19,698 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:19,698 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:19,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:19,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:19,698 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:19,698 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:19,698 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,698 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:19,698 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:19,698 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:19,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:19,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:19,715 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/e21325e9850c40e5b6f5834112b35855 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/e21325e9850c40e5b6f5834112b35855 2023-04-09 09:56:19,719 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c8127616cf5355dd7f1db99764192b95/B of c8127616cf5355dd7f1db99764192b95 into e21325e9850c40e5b6f5834112b35855(size=12.8 K), total size for store is 24.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:56:19,719 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c8127616cf5355dd7f1db99764192b95: 2023-04-09 09:56:19,719 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95., storeName=c8127616cf5355dd7f1db99764192b95/B, priority=13, startTime=1681034179294; duration=0sec 2023-04-09 09:56:19,719 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:56:19,798 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:19,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:19,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:19,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:19,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:19,804 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:19,804 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:19,804 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:19,804 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:19,804 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:19,805 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:19,806 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:19,806 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:19,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:19,806 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,806 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:19,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:19,807 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:19,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:19,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:19,807 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:19,807 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:19,807 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:19,807 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:19,807 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:19,808 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:19,808 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:19,808 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:19,808 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:19,808 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:19,808 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:19,808 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:19,808 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:19,808 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,808 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:19,808 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:19,810 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:19,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:19,810 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:19,810 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:19,810 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:19,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:19,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:19,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:19,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:19,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:19,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@133d7c7[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:19,812 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:19,812 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:19,814 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:19,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:19,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:19,814 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:19,814 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:19,814 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:19,815 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,815 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:19,816 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,816 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:19,816 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:19,816 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:19,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:19,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:19,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:19,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:19,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:19,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:19,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:19,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:19,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:19,821 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:19,821 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:19,821 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:19,821 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:19,821 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:19,822 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:19,823 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:19,823 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:19,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:19,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:19,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:19,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:19,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:19,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:19,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:19,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:19,823 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:19,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:19,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:19,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:19,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,825 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:19,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:19,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,827 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:19,827 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,827 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:19,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:19,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:19,827 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:19,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:19,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:19,827 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:19,827 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:56:19,828 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:19,828 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,828 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:19,828 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:19,828 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:19,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:19,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:19,828 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:19,828 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:19,865 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:19,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60448 deadline: 1681034239863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,865 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:19,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60436 deadline: 1681034239863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,866 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:19,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60460 deadline: 1681034239864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,866 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:19,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60506 deadline: 1681034239865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,869 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:19,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60498 deadline: 1681034239867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,922 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=599, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/A/d30d46ac14f24252ad31aebdad140b0d 2023-04-09 09:56:19,928 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:19,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:19,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:19,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:19,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:19,936 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:19,936 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:19,936 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:19,936 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:19,937 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:19,937 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:19,941 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:19,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:19,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:19,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:19,941 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,942 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:19,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:19,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:19,942 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:19,942 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,942 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:19,943 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:19,943 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:19,943 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:19,943 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:19,943 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:19,944 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:19,944 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:19,944 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:19,944 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:19,944 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:19,944 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=599 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/71e2dc7c4e3f4b5b8d86bbcd1794b24a 2023-04-09 09:56:19,944 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:19,945 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:19,945 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:19,945 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,945 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:19,945 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:19,946 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:19,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:19,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:19,947 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:19,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:19,948 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:19,948 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:19,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:19,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:19,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:19,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f410577[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:19,949 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:19,949 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:19,950 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:19,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:19,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:19,951 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:19,951 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:19,951 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:19,951 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,951 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:19,952 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,952 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:19,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:19,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:19,952 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:19,952 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:19,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:19,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:19,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:19,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:19,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:19,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:19,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:19,957 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:19,957 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:19,957 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:19,957 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:19,957 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:19,957 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=599 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/C/396aa1b0546f4fc8bd9b798c1ac99e7c 2023-04-09 09:56:19,960 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:19,960 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:19,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:19,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:19,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:19,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:19,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:19,960 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:19,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:19,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:19,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:19,961 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,961 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/A/d30d46ac14f24252ad31aebdad140b0d as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/d30d46ac14f24252ad31aebdad140b0d 2023-04-09 09:56:19,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:19,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:19,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:19,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:19,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,962 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:19,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:19,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,964 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/d30d46ac14f24252ad31aebdad140b0d, entries=350, sequenceid=599, filesize=52.9 K 2023-04-09 09:56:19,964 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,964 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:19,964 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:19,964 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:19,965 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:19,965 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:19,965 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:56:19,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:19,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:19,965 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:19,965 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:19,965 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:19,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:19,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:19,965 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:19,965 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:19,965 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:19,965 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/71e2dc7c4e3f4b5b8d86bbcd1794b24a as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/71e2dc7c4e3f4b5b8d86bbcd1794b24a 2023-04-09 09:56:19,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:19,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:19,968 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/71e2dc7c4e3f4b5b8d86bbcd1794b24a, entries=100, sequenceid=599, filesize=9.5 K 2023-04-09 09:56:19,969 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/C/396aa1b0546f4fc8bd9b798c1ac99e7c as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/396aa1b0546f4fc8bd9b798c1ac99e7c 2023-04-09 09:56:19,972 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/396aa1b0546f4fc8bd9b798c1ac99e7c, entries=100, sequenceid=599, filesize=9.5 K 2023-04-09 09:56:19,973 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for c8127616cf5355dd7f1db99764192b95 in 484ms, sequenceid=599, compaction requested=true 2023-04-09 09:56:19,973 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c8127616cf5355dd7f1db99764192b95: 2023-04-09 09:56:19,973 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:56:19,973 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:56:19,973 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-09 09:56:19,973 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:56:19,973 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-09 09:56:19,974 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 107188 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-09 09:56:19,974 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1912): c8127616cf5355dd7f1db99764192b95/A is initiating minor compaction (all files) 2023-04-09 09:56:19,974 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35034 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-09 09:56:19,974 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c8127616cf5355dd7f1db99764192b95/A in TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:19,974 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): c8127616cf5355dd7f1db99764192b95/B is initiating minor compaction (all files) 2023-04-09 09:56:19,974 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/f2527b9174e74bf99667a59e5f75d544, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/b127ecd481d54190b3f45c859710248a, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/d30d46ac14f24252ad31aebdad140b0d] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp, totalSize=104.7 K 2023-04-09 09:56:19,974 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c8127616cf5355dd7f1db99764192b95/B in TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:19,974 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/e21325e9850c40e5b6f5834112b35855, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/b86f3364105b47fdad86f9c1e2616c99, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/71e2dc7c4e3f4b5b8d86bbcd1794b24a] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp, totalSize=34.2 K 2023-04-09 09:56:19,974 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting f2527b9174e74bf99667a59e5f75d544, keycount=150, bloomtype=ROW, size=26.3 K, encoding=NONE, compression=NONE, seqNum=548, earliestPutTs=1681034178797 2023-04-09 09:56:19,974 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting e21325e9850c40e5b6f5834112b35855, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=548, earliestPutTs=1681034178797 2023-04-09 09:56:19,975 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting b127ecd481d54190b3f45c859710248a, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=586, earliestPutTs=1681034178867 2023-04-09 09:56:19,975 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting b86f3364105b47fdad86f9c1e2616c99, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=586, earliestPutTs=1681034178867 2023-04-09 09:56:19,975 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting d30d46ac14f24252ad31aebdad140b0d, keycount=350, bloomtype=ROW, size=52.9 K, encoding=NONE, compression=NONE, seqNum=599, earliestPutTs=1681034179469 2023-04-09 09:56:19,975 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 71e2dc7c4e3f4b5b8d86bbcd1794b24a, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=599, earliestPutTs=1681034179488 2023-04-09 09:56:19,980 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): c8127616cf5355dd7f1db99764192b95#B#compaction#642 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:56:20,001 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] throttle.PressureAwareThroughputController(145): c8127616cf5355dd7f1db99764192b95#A#compaction#643 average throughput is 0.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:56:20,007 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/f32324d96c0d4c61a905cdb87b12cd69 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/f32324d96c0d4c61a905cdb87b12cd69 2023-04-09 09:56:20,011 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c8127616cf5355dd7f1db99764192b95/B of c8127616cf5355dd7f1db99764192b95 into f32324d96c0d4c61a905cdb87b12cd69(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:56:20,011 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c8127616cf5355dd7f1db99764192b95: 2023-04-09 09:56:20,011 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95., storeName=c8127616cf5355dd7f1db99764192b95/B, priority=13, startTime=1681034179973; duration=0sec 2023-04-09 09:56:20,011 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:56:20,012 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-09 09:56:20,012 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34966 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-09 09:56:20,013 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): c8127616cf5355dd7f1db99764192b95/C is initiating minor compaction (all files) 2023-04-09 09:56:20,013 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c8127616cf5355dd7f1db99764192b95/C in TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:20,013 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/4a0d633a6ca14844925fd937da30fc1e, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/365ea5ede1e94dada6f1176d0c6190f3, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/396aa1b0546f4fc8bd9b798c1ac99e7c] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp, totalSize=34.1 K 2023-04-09 09:56:20,013 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 4a0d633a6ca14844925fd937da30fc1e, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=548, earliestPutTs=1681034178797 2023-04-09 09:56:20,013 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 365ea5ede1e94dada6f1176d0c6190f3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=586, earliestPutTs=1681034178867 2023-04-09 09:56:20,014 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 396aa1b0546f4fc8bd9b798c1ac99e7c, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=599, earliestPutTs=1681034179488 2023-04-09 09:56:20,020 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): c8127616cf5355dd7f1db99764192b95#C#compaction#644 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:56:20,039 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/C/77aaf68bce0a4a39b39aba633af90cf2 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/77aaf68bce0a4a39b39aba633af90cf2 2023-04-09 09:56:20,047 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c8127616cf5355dd7f1db99764192b95/C of c8127616cf5355dd7f1db99764192b95 into 77aaf68bce0a4a39b39aba633af90cf2(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:56:20,047 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c8127616cf5355dd7f1db99764192b95: 2023-04-09 09:56:20,047 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95., storeName=c8127616cf5355dd7f1db99764192b95/C, priority=13, startTime=1681034179973; duration=0sec 2023-04-09 09:56:20,047 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:56:20,053 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/A/a20c35cbcad54e808fa506cb7a5be7e4 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/a20c35cbcad54e808fa506cb7a5be7e4 2023-04-09 09:56:20,057 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c8127616cf5355dd7f1db99764192b95/A of c8127616cf5355dd7f1db99764192b95 into a20c35cbcad54e808fa506cb7a5be7e4(size=26.4 K), total size for store is 26.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:56:20,057 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c8127616cf5355dd7f1db99764192b95: 2023-04-09 09:56:20,057 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95., storeName=c8127616cf5355dd7f1db99764192b95/A, priority=13, startTime=1681034179973; duration=0sec 2023-04-09 09:56:20,058 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:56:20,065 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:20,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:20,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:20,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:20,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:20,076 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:20,076 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:20,076 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:20,076 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:20,076 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:20,077 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:20,079 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:20,079 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:20,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:20,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:20,079 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,079 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:20,080 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:20,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:20,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:20,080 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:20,081 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:20,081 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:20,081 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:20,081 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:20,081 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:20,088 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:20,088 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:20,088 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:20,088 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:20,088 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c8127616cf5355dd7f1db99764192b95 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-09 09:56:20,088 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=A 2023-04-09 09:56:20,089 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:56:20,089 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=B 2023-04-09 09:56:20,089 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:56:20,089 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=C 2023-04-09 09:56:20,089 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:56:20,105 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230409ba43f49e6cc34dab892436ae1591564c to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409ba43f49e6cc34dab892436ae1591564c 2023-04-09 09:56:20,110 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=639, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/A/0833cffa55de47b59ea6266b04804399 2023-04-09 09:56:20,121 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=639 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/5ed522c8593044e5b66dc2e2ba59b365 2023-04-09 09:56:20,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on c8127616cf5355dd7f1db99764192b95 2023-04-09 09:56:20,167 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:20,175 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:20,176 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:20,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60506 deadline: 1681034240172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60498 deadline: 1681034240172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,176 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:20,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60460 deadline: 1681034240172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,178 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:20,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60448 deadline: 1681034240175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,178 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:20,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60436 deadline: 1681034240176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,278 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:20,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60506 deadline: 1681034240277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,278 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:20,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60460 deadline: 1681034240277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,280 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:20,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60448 deadline: 1681034240279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,280 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:20,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60436 deadline: 1681034240279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,482 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:20,482 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:20,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60460 deadline: 1681034240480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60506 deadline: 1681034240480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,482 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:20,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60436 deadline: 1681034240481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,482 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:20,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60448 deadline: 1681034240482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,531 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=639 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/C/a9b92a2e33a444039ee2d96f32e73b7a 2023-04-09 09:56:20,534 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/A/0833cffa55de47b59ea6266b04804399 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/0833cffa55de47b59ea6266b04804399 2023-04-09 09:56:20,538 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/0833cffa55de47b59ea6266b04804399, entries=150, sequenceid=639, filesize=25.4 K 2023-04-09 09:56:20,538 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/5ed522c8593044e5b66dc2e2ba59b365 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/5ed522c8593044e5b66dc2e2ba59b365 2023-04-09 09:56:20,541 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/5ed522c8593044e5b66dc2e2ba59b365, entries=150, sequenceid=639, filesize=11.9 K 2023-04-09 09:56:20,542 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/C/a9b92a2e33a444039ee2d96f32e73b7a as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/a9b92a2e33a444039ee2d96f32e73b7a 2023-04-09 09:56:20,545 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/a9b92a2e33a444039ee2d96f32e73b7a, entries=150, sequenceid=639, filesize=11.9 K 2023-04-09 09:56:20,545 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for c8127616cf5355dd7f1db99764192b95 in 457ms, sequenceid=639, compaction requested=false 2023-04-09 09:56:20,546 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c8127616cf5355dd7f1db99764192b95: 2023-04-09 09:56:20,546 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:20,546 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:20,546 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:20,546 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,546 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:20,546 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:20,547 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,547 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:20,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:20,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:20,548 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:20,548 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:20,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:20,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:20,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:20,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:20,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:20,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37e1b408[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:20,552 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:20,552 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:20,555 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:20,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:20,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:20,555 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,555 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:20,563 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:20,563 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:20,563 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:20,567 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:20,567 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,567 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:20,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:20,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:20,567 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:20,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:20,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:20,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:20,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:20,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:20,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:20,570 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:20,570 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:20,570 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:20,570 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:20,570 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:20,570 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:20,572 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:20,572 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:20,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:20,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:20,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:20,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:20,572 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:20,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:20,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:20,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:20,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:20,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:20,573 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:20,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:20,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:20,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,574 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:20,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:20,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,577 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:20,578 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:20,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:20,578 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:20,578 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:20,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:20,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:20,578 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:20,578 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:20,578 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,578 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:20,578 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:20,578 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:20,578 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:20,578 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:20,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:20,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:20,678 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:20,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:20,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on c8127616cf5355dd7f1db99764192b95 2023-04-09 09:56:20,680 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c8127616cf5355dd7f1db99764192b95 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-09 09:56:20,680 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=A 2023-04-09 09:56:20,680 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:56:20,680 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=B 2023-04-09 09:56:20,680 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:56:20,680 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=C 2023-04-09 09:56:20,680 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:56:20,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:20,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:20,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:20,682 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:20,682 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:20,682 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:20,682 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:20,683 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:20,683 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:20,684 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:20,684 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:20,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:20,684 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,684 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:20,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:20,684 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:20,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:20,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:20,685 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:20,689 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:20,689 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:20,689 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:20,689 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:20,689 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:20,690 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:20,690 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:20,690 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:20,690 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:20,690 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:20,690 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:20,690 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:20,690 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:20,690 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,690 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:20,690 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:20,692 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:20,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:20,692 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:20,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:20,692 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:20,693 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:20,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:20,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:20,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:20,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:20,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2879e1b4[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:20,694 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:20,694 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:20,695 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:20,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:20,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:20,695 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:20,695 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:20,695 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:20,695 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,695 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:20,697 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:20,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:20,697 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:20,697 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:20,697 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:20,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:20,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:20,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:20,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,701 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230409b030095902e545f69a6a20aaa41cbf5d to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409b030095902e545f69a6a20aaa41cbf5d 2023-04-09 09:56:20,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:20,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:20,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:20,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:20,704 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:20,704 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:20,704 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:20,704 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:20,704 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:20,705 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:20,705 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:20,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:20,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:20,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:20,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:20,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:20,705 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:20,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:20,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:20,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:20,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:20,706 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:20,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:20,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:20,707 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:20,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:20,711 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:20,711 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:20,711 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:20,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:20,711 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:20,711 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:20,711 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,711 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:20,711 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:20,711 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:20,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:20,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:20,712 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:20,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:20,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:20,712 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=651, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/A/58b11c680a3445998a63e07d66d47e2d 2023-04-09 09:56:20,712 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:20,712 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:20,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,717 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:56:20,742 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:20,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60498 deadline: 1681034240740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,784 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:20,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60460 deadline: 1681034240783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,785 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:20,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60436 deadline: 1681034240784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,787 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:20,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60448 deadline: 1681034240784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,787 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:20,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60506 deadline: 1681034240784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,812 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:20,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:20,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:20,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:20,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:20,829 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:20,829 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:20,829 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:20,829 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:20,829 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:20,829 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:20,834 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:20,834 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:20,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:20,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:20,835 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,835 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:20,835 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:20,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:20,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:20,835 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:20,836 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:20,836 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:20,836 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:20,836 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:20,836 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:20,844 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:20,844 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:20,844 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:20,844 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:20,844 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:20,844 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:20,844 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:20,844 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:20,844 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,844 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:20,844 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:20,844 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:20,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60498 deadline: 1681034240843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,846 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,846 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:20,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:20,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:20,846 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:20,846 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:20,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:20,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:20,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:20,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:20,848 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:20,848 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37eb8693[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:20,848 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:20,849 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:20,850 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:20,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:20,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:20,850 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:20,850 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:20,850 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:20,850 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,850 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:20,851 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:20,851 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:20,851 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,851 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:20,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:20,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:20,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:20,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:20,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:20,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:20,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:20,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:20,856 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:20,856 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:20,856 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:20,856 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:20,856 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:20,856 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:20,859 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:20,859 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:20,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:20,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:20,859 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:20,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:20,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:20,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:20,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:20,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:20,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:20,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:20,861 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:20,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:20,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:20,862 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:20,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:20,865 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:20,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:20,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:20,865 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,865 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:20,865 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:20,865 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,865 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:20,865 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:20,865 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:20,865 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:20,865 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:20,865 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:20,865 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:20,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:20,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:20,865 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:56:20,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:20,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:20,965 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:20,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:20,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:20,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:20,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:20,970 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:20,970 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:20,970 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:20,970 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:20,970 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:20,970 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:20,976 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:20,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:20,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:20,976 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:20,977 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,977 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:20,977 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:20,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:20,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:20,977 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:20,978 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:20,978 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:20,978 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:20,978 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:20,978 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:20,980 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:20,980 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:20,980 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:20,980 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:20,980 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:20,980 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:20,980 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:20,980 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:20,980 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,980 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:20,981 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:20,982 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,982 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:20,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:20,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:20,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:20,982 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:20,982 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:20,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:20,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:20,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:20,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:20,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@407657a4[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:20,984 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:20,984 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:20,985 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:20,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:20,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:20,985 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:20,985 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:20,985 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:20,985 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,985 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:20,986 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,986 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:20,986 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:20,986 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:20,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:20,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:20,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:20,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:20,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:20,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:20,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:20,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:20,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:20,989 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:20,989 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:20,989 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:20,989 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:20,989 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:20,991 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:20,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:20,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:20,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:20,991 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:20,991 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:20,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:20,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:20,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:20,992 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:20,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:20,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:20,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:20,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:20,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:20,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:20,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:20,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,994 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:20,999 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:20,999 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:21,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:21,000 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:21,000 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:21,000 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:21,000 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,000 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:21,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:21,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:21,000 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:21,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:21,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:21,000 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:21,000 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:21,000 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:21,000 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:21,049 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:21,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60498 deadline: 1681034241047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,100 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:21,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:21,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:21,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:21,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:21,104 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:21,104 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:21,104 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:21,104 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:21,104 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:21,104 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:21,118 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:21,119 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:21,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:21,119 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:21,119 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:21,119 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:21,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:21,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:21,120 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:21,120 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:21,120 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:21,120 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:21,120 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:21,120 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:21,120 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=651 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/bd80be54b5e64f308b2ae26f755e3bf6 2023-04-09 09:56:21,120 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:21,121 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:21,120 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:21,121 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:21,121 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:21,121 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:21,121 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:21,121 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:21,121 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,121 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:21,121 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:21,124 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:21,124 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:21,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:21,124 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:21,124 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:21,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:21,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:21,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:21,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:21,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:21,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a8dfa56[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:21,126 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:21,126 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:21,127 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:21,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:21,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:21,127 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,128 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:21,127 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:21,128 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:21,128 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:21,130 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,130 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:21,130 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:21,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:21,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:21,130 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:21,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:21,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:21,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:21,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:21,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:21,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:21,132 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:21,132 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:21,132 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:21,132 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:21,132 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:21,133 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:21,134 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:21,134 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:21,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:21,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:21,134 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:21,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:21,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:21,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:21,134 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:21,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:21,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:21,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:21,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:21,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:21,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:21,136 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:21,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:21,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,138 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,138 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:21,138 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:21,138 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:21,138 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:56:21,138 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:21,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:21,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:21,138 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:21,138 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:21,138 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:21,138 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,138 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:21,138 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:21,138 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:21,138 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:21,138 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:21,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:21,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:21,239 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:21,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:21,241 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:21,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:21,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:21,242 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:21,242 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:21,242 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:21,242 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:21,242 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:21,242 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:21,243 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,243 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:21,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:21,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:21,244 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,244 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:21,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:21,244 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:21,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:21,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:21,244 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:21,244 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:21,245 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:21,245 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:21,245 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:21,245 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:21,245 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:21,245 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:21,245 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:21,245 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:21,245 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:21,245 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:21,245 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:21,245 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:21,245 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,245 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:21,245 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:21,247 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,247 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:21,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:21,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:21,248 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:21,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:21,248 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:21,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:21,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:21,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:21,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:21,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d1ec0f2[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:21,249 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:21,249 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:21,250 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:21,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:21,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:21,250 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:21,250 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:21,250 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,250 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:21,250 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:21,251 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,251 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:21,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:21,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:21,252 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:21,252 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:21,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:21,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:21,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:21,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:21,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:21,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:21,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:21,254 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:21,254 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:21,254 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:21,254 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:21,254 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:21,255 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:21,255 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:21,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:21,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:21,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:21,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:21,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:21,255 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:21,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:21,256 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:21,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:21,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:21,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:21,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:21,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:21,257 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:21,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:21,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,260 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,260 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:21,260 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:21,260 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:21,260 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:56:21,260 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:21,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:21,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:21,260 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,261 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:21,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:21,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:21,261 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:21,261 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:21,261 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:21,261 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:21,261 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:21,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:21,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:21,288 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:21,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60436 deadline: 1681034241287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,291 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:21,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60460 deadline: 1681034241289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,293 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:21,293 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:21,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60448 deadline: 1681034241292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60506 deadline: 1681034241292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,350 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:21,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60498 deadline: 1681034241350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,361 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:21,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:21,364 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:21,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:21,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:21,365 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:21,365 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:21,365 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:21,366 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:21,366 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:21,366 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:21,367 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,368 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:21,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:21,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:21,368 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,369 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:21,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:21,369 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:21,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:21,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:21,370 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:21,370 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:21,370 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:21,370 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:21,370 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:21,370 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:21,371 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:21,371 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:21,371 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:21,371 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:21,371 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:21,371 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:21,371 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:21,371 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:21,371 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,371 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:21,371 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:21,373 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,373 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:21,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:21,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:21,374 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:21,374 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:21,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:21,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:21,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:21,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:21,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:21,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76aa12dc[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:21,375 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:21,375 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:21,377 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,377 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:21,379 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:21,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:21,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:21,379 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:21,379 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:21,379 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:21,382 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,382 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:21,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:21,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:21,382 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:21,382 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:21,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:21,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:21,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:21,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:21,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:21,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:21,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:21,387 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:21,387 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:21,387 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:21,387 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:21,387 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:21,389 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:21,389 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:21,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:21,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:21,389 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:21,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:21,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:21,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:21,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:21,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:21,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:21,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:21,391 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:21,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:21,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:21,394 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:21,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:21,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,398 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,398 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:21,398 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:21,398 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:21,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:21,398 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:21,398 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,398 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:21,398 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:21,398 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:21,398 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:21,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:21,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:21,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:21,399 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:21,399 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:21,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:21,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:21,499 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:21,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:21,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:21,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:21,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:21,502 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:21,502 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:21,502 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:21,502 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:21,502 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:21,502 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:21,503 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,503 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:21,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:21,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:21,504 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,504 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:21,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:21,504 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:21,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:21,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:21,504 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:21,505 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:21,505 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:21,505 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:21,505 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:21,505 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:21,505 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:21,505 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:21,505 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:21,505 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:21,505 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:21,505 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:21,505 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:21,505 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:21,505 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,505 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:21,505 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:21,507 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:21,507 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:21,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:21,507 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:21,507 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:21,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:21,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:21,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:21,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:21,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:21,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@413b0da[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:21,508 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:21,508 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:21,509 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:21,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:21,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:21,509 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:21,509 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:21,509 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:21,510 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,510 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:21,512 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:21,512 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:21,512 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,512 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:21,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:21,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:21,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:21,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:21,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:21,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:21,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:21,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:21,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:21,514 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:21,514 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:21,514 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:21,514 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:21,514 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:21,516 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:21,516 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:21,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:21,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:21,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:21,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:21,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:21,516 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:21,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:21,516 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:21,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:21,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:21,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:21,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:21,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:21,517 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:21,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:21,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,520 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,520 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:21,520 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:21,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:21,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:21,520 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:21,520 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:21,520 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,520 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:21,520 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:21,520 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:21,520 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:56:21,520 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:21,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:21,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:21,520 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:21,520 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:21,521 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:21,521 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:21,530 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=651 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/C/b02c9efc2be04ae3a5762c35d56962fe 2023-04-09 09:56:21,533 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/A/58b11c680a3445998a63e07d66d47e2d as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/58b11c680a3445998a63e07d66d47e2d 2023-04-09 09:56:21,536 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/58b11c680a3445998a63e07d66d47e2d, entries=200, sequenceid=651, filesize=32.3 K 2023-04-09 09:56:21,536 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/bd80be54b5e64f308b2ae26f755e3bf6 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/bd80be54b5e64f308b2ae26f755e3bf6 2023-04-09 09:56:21,539 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/bd80be54b5e64f308b2ae26f755e3bf6, entries=150, sequenceid=651, filesize=11.9 K 2023-04-09 09:56:21,539 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/C/b02c9efc2be04ae3a5762c35d56962fe as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/b02c9efc2be04ae3a5762c35d56962fe 2023-04-09 09:56:21,542 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/b02c9efc2be04ae3a5762c35d56962fe, entries=150, sequenceid=651, filesize=11.9 K 2023-04-09 09:56:21,543 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for c8127616cf5355dd7f1db99764192b95 in 863ms, sequenceid=651, compaction requested=true 2023-04-09 09:56:21,543 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c8127616cf5355dd7f1db99764192b95: 2023-04-09 09:56:21,543 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:56:21,543 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-09 09:56:21,543 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-09 09:56:21,543 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:56:21,543 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:56:21,544 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 86166 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-09 09:56:21,544 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37580 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-09 09:56:21,544 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1912): c8127616cf5355dd7f1db99764192b95/A is initiating minor compaction (all files) 2023-04-09 09:56:21,544 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): c8127616cf5355dd7f1db99764192b95/B is initiating minor compaction (all files) 2023-04-09 09:56:21,544 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c8127616cf5355dd7f1db99764192b95/A in TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:21,544 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c8127616cf5355dd7f1db99764192b95/B in TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:21,544 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/a20c35cbcad54e808fa506cb7a5be7e4, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/0833cffa55de47b59ea6266b04804399, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/58b11c680a3445998a63e07d66d47e2d] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp, totalSize=84.1 K 2023-04-09 09:56:21,544 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/f32324d96c0d4c61a905cdb87b12cd69, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/5ed522c8593044e5b66dc2e2ba59b365, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/bd80be54b5e64f308b2ae26f755e3bf6] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp, totalSize=36.7 K 2023-04-09 09:56:21,545 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting a20c35cbcad54e808fa506cb7a5be7e4, keycount=150, bloomtype=ROW, size=26.4 K, encoding=NONE, compression=NONE, seqNum=599, earliestPutTs=1681034178881 2023-04-09 09:56:21,545 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting f32324d96c0d4c61a905cdb87b12cd69, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=599, earliestPutTs=1681034178881 2023-04-09 09:56:21,545 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 0833cffa55de47b59ea6266b04804399, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=639, earliestPutTs=1681034179547 2023-04-09 09:56:21,545 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 5ed522c8593044e5b66dc2e2ba59b365, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=639, earliestPutTs=1681034179547 2023-04-09 09:56:21,545 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting bd80be54b5e64f308b2ae26f755e3bf6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=651, earliestPutTs=1681034180170 2023-04-09 09:56:21,546 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 58b11c680a3445998a63e07d66d47e2d, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=651, earliestPutTs=1681034180170 2023-04-09 09:56:21,551 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): c8127616cf5355dd7f1db99764192b95#B#compaction#651 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:56:21,569 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] throttle.PressureAwareThroughputController(145): c8127616cf5355dd7f1db99764192b95#A#compaction#652 average throughput is 1.04 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:56:21,589 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/A/4ea33ebfce6b49ec88127a20a3322cb0 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/4ea33ebfce6b49ec88127a20a3322cb0 2023-04-09 09:56:21,593 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c8127616cf5355dd7f1db99764192b95/A of c8127616cf5355dd7f1db99764192b95 into 4ea33ebfce6b49ec88127a20a3322cb0(size=26.5 K), total size for store is 26.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:56:21,593 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c8127616cf5355dd7f1db99764192b95: 2023-04-09 09:56:21,593 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95., storeName=c8127616cf5355dd7f1db99764192b95/A, priority=13, startTime=1681034181543; duration=0sec 2023-04-09 09:56:21,593 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:56:21,593 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-09 09:56:21,594 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37512 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-09 09:56:21,594 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1912): c8127616cf5355dd7f1db99764192b95/C is initiating minor compaction (all files) 2023-04-09 09:56:21,594 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c8127616cf5355dd7f1db99764192b95/C in TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:21,594 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/77aaf68bce0a4a39b39aba633af90cf2, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/a9b92a2e33a444039ee2d96f32e73b7a, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/b02c9efc2be04ae3a5762c35d56962fe] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp, totalSize=36.6 K 2023-04-09 09:56:21,594 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 77aaf68bce0a4a39b39aba633af90cf2, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=599, earliestPutTs=1681034178881 2023-04-09 09:56:21,595 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting a9b92a2e33a444039ee2d96f32e73b7a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=639, earliestPutTs=1681034179547 2023-04-09 09:56:21,595 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting b02c9efc2be04ae3a5762c35d56962fe, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=651, earliestPutTs=1681034180170 2023-04-09 09:56:21,599 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] throttle.PressureAwareThroughputController(145): c8127616cf5355dd7f1db99764192b95#C#compaction#653 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:56:21,606 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/C/b746c6c37ca249979c59d4f35a07b458 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/b746c6c37ca249979c59d4f35a07b458 2023-04-09 09:56:21,611 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c8127616cf5355dd7f1db99764192b95/C of c8127616cf5355dd7f1db99764192b95 into b746c6c37ca249979c59d4f35a07b458(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:56:21,611 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c8127616cf5355dd7f1db99764192b95: 2023-04-09 09:56:21,611 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95., storeName=c8127616cf5355dd7f1db99764192b95/C, priority=13, startTime=1681034181543; duration=0sec 2023-04-09 09:56:21,611 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:56:21,621 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:21,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:21,623 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:21,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:21,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:21,624 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:21,624 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:21,624 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:21,624 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:21,624 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:21,624 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:21,625 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,625 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:21,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:21,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:21,626 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:21,626 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:21,626 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:21,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:21,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:21,627 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:21,627 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:21,627 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:21,627 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:21,627 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:21,627 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:21,627 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:21,627 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:21,627 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:21,627 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:21,627 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c8127616cf5355dd7f1db99764192b95 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-09 09:56:21,628 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=A 2023-04-09 09:56:21,628 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:56:21,628 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=B 2023-04-09 09:56:21,628 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:56:21,628 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=C 2023-04-09 09:56:21,628 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:56:21,640 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304092671355463a64ebe87903c47c83f8b4b to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304092671355463a64ebe87903c47c83f8b4b 2023-04-09 09:56:21,644 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=673, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/A/72f6462c665845c0b3e5fe2be869c43c 2023-04-09 09:56:21,654 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=673 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/36b8434a72f24a188893c8738e997c94 2023-04-09 09:56:21,670 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=673 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/C/8f5a2dce2ec74cb880aedf8b2feef72b 2023-04-09 09:56:21,673 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/A/72f6462c665845c0b3e5fe2be869c43c as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/72f6462c665845c0b3e5fe2be869c43c 2023-04-09 09:56:21,680 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/72f6462c665845c0b3e5fe2be869c43c, entries=150, sequenceid=673, filesize=25.4 K 2023-04-09 09:56:21,681 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/36b8434a72f24a188893c8738e997c94 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/36b8434a72f24a188893c8738e997c94 2023-04-09 09:56:21,685 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/36b8434a72f24a188893c8738e997c94, entries=150, sequenceid=673, filesize=11.9 K 2023-04-09 09:56:21,686 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/C/8f5a2dce2ec74cb880aedf8b2feef72b as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/8f5a2dce2ec74cb880aedf8b2feef72b 2023-04-09 09:56:21,689 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/8f5a2dce2ec74cb880aedf8b2feef72b, entries=150, sequenceid=673, filesize=11.9 K 2023-04-09 09:56:21,690 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for c8127616cf5355dd7f1db99764192b95 in 63ms, sequenceid=673, compaction requested=false 2023-04-09 09:56:21,690 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c8127616cf5355dd7f1db99764192b95: 2023-04-09 09:56:21,690 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:21,690 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:21,690 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:21,690 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,690 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:21,690 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:21,692 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:21,692 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:21,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:21,693 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:21,693 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:21,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:21,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:21,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:21,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:21,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:21,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1159b544[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:21,696 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:21,696 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:21,697 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:21,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:21,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:21,697 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:21,697 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:21,697 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:21,697 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,697 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:21,698 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:21,698 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:21,698 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:21,698 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:21,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:21,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:21,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:21,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:21,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:21,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:21,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:21,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:21,701 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:21,702 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:21,702 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:21,702 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:21,702 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:21,705 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:21,705 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:21,705 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:21,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:21,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:21,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:21,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:21,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:21,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:21,706 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:21,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:21,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:21,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:21,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:21,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:21,707 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:21,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:21,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,709 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,710 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:21,710 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:21,710 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,710 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:21,710 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:21,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:21,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:21,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:21,710 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:21,710 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:21,710 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:21,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:21,710 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:56:21,710 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:21,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:21,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:21,710 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:21,710 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:21,810 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:21,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:21,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:21,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:21,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:21,815 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:21,815 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:21,815 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:21,815 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:21,815 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:21,815 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:21,817 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:21,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:21,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:21,817 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:21,817 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,817 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:21,817 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:21,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:21,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:21,818 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:21,818 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:21,818 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:21,818 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:21,818 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:21,818 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:21,818 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:21,819 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:21,819 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:21,819 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c8127616cf5355dd7f1db99764192b95: 2023-04-09 09:56:21,819 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:21,819 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:21,819 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:21,819 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,819 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:21,819 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:21,821 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,821 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:21,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:21,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:21,822 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:21,822 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:21,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:21,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:21,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:21,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:21,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:21,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a2beecd[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:21,824 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:21,824 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:21,833 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:21,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:21,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:21,833 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:21,833 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:21,833 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:21,833 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,833 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:21,834 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:21,834 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,834 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:21,834 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:21,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:21,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:21,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:21,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:21,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:21,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:21,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:21,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:21,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:21,838 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:21,838 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:21,838 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:21,838 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:21,838 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:21,839 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:21,839 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:21,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:21,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:21,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:21,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:21,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:21,839 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:21,840 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:21,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:21,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:21,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:21,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:21,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:21,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:21,841 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:21,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:21,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,844 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,844 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:21,844 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:21,844 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:21,844 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:21,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:21,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:21,844 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:21,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:21,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:21,844 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,845 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:21,845 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:21,845 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:21,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:21,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:21,845 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:21,845 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:21,845 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:56:21,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on c8127616cf5355dd7f1db99764192b95 2023-04-09 09:56:21,886 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c8127616cf5355dd7f1db99764192b95 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-09 09:56:21,886 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=A 2023-04-09 09:56:21,886 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:56:21,886 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=B 2023-04-09 09:56:21,886 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:56:21,886 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=C 2023-04-09 09:56:21,886 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:56:21,899 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304098ddbb6ff68cc46beaf026d94d4554d7f to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304098ddbb6ff68cc46beaf026d94d4554d7f 2023-04-09 09:56:21,916 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=684, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/A/e932dcb703b3404aaf865d9d31eb8e1a 2023-04-09 09:56:21,945 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:21,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:21,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:21,949 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034181930, max timestamp=1681034181946Num uniques 859; 2023-04-09 09:56:21,949 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:56:21,949 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034181930, max timestamp=1681034181946Num uniques 859; 2023-04-09 09:56:21,949 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034181930, max timestamp=1681034181946Num uniques 859; 2023-04-09 09:56:21,949 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:56:21,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:21,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:21,950 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:21,950 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:21,950 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:21,950 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:21,950 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:21,950 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:21,954 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:21,954 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:21,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:21,955 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,955 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:21,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:21,955 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:21,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:21,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:21,956 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:21,957 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:21,957 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:21,957 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:21,957 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:21,957 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:21,949 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:56:21,961 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:21,961 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:21,961 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:21,962 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:21,962 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:21,962 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:21,962 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:21,962 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:21,962 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,962 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:21,962 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:21,963 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,963 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:21,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:21,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:21,964 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:21,964 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:21,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:21,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:21,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:21,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:21,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:21,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5cb13fa6[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:21,966 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:21,966 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:21,967 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:21,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:21,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:21,968 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:21,968 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:21,968 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:21,968 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,968 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:21,969 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/733f3b3ea567457c96f0d4f49461378c as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/733f3b3ea567457c96f0d4f49461378c 2023-04-09 09:56:21,970 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,970 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:21,970 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:21,970 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:21,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:21,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:21,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:21,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:21,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:21,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:21,973 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c8127616cf5355dd7f1db99764192b95/B of c8127616cf5355dd7f1db99764192b95 into 733f3b3ea567457c96f0d4f49461378c(size=13.0 K), total size for store is 24.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:56:21,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:21,973 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c8127616cf5355dd7f1db99764192b95: 2023-04-09 09:56:21,973 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95., storeName=c8127616cf5355dd7f1db99764192b95/B, priority=13, startTime=1681034181543; duration=0sec 2023-04-09 09:56:21,974 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:56:21,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:21,974 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:21,974 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:21,974 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:21,974 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:21,974 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:21,974 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:21,976 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:21,976 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:21,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:21,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:21,976 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:21,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:21,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:21,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:21,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:21,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:21,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:21,977 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:21,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:21,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:21,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:21,978 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:21,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:21,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,982 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:21,982 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:21,982 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,982 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:21,982 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:21,982 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:21,982 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:21,982 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:21,982 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:21,982 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:21,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:21,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:21,983 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:21,983 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:21,983 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:56:21,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:21,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:21,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:21,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:22,016 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:22,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60498 deadline: 1681034242012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,083 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:22,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:22,085 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:22,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:22,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:22,086 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:22,086 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:22,086 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:22,086 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:22,086 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:22,086 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:22,088 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:22,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:22,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:22,088 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:22,088 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,088 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:22,089 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:22,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:22,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:22,089 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:22,089 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:22,089 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:22,089 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:22,089 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:22,089 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:22,092 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:22,092 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:22,092 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:22,092 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:22,092 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:22,092 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:22,092 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:22,092 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:22,092 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,092 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:22,092 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:22,095 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,095 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:22,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:22,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:22,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:22,096 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:22,096 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:22,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:22,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:22,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:22,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:22,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@453dc18a[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:22,097 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:22,097 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:22,098 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:22,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:22,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:22,098 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:22,098 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:22,098 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:22,098 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,098 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:22,099 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,099 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:22,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:22,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:22,099 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:22,099 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:22,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:22,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:22,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:22,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:22,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:22,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:22,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:22,101 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:22,101 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:22,101 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:22,101 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:22,101 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:22,102 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:22,102 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:22,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:22,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:22,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:22,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:22,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:22,102 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:22,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:22,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:22,103 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:22,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:22,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:22,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:22,104 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:22,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:22,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:22,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,108 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:22,108 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:22,108 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:22,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:22,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:22,108 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:22,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:22,108 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:22,108 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:22,108 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,108 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:22,108 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:22,108 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:22,108 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:22,108 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:56:22,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:22,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:22,108 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:22,118 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:22,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60498 deadline: 1681034242117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,208 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:22,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:22,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:22,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:22,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:22,216 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:22,216 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:22,216 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:22,216 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:22,217 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:22,217 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:22,219 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:22,219 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:22,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:22,220 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,220 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:22,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:22,220 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:22,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:22,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:22,221 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:22,221 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:22,221 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:22,221 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:22,221 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:22,221 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:22,228 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:22,228 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:22,228 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:22,228 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:22,228 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:22,228 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:22,228 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:22,228 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:22,228 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,228 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:22,228 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:22,230 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,230 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:22,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:22,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:22,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:22,230 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:22,231 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:22,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:22,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:22,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:22,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:22,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1992affc[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:22,232 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:22,232 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:22,233 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:22,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:22,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:22,233 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:22,233 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:22,233 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:22,233 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,233 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:22,234 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:22,234 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:22,234 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:22,235 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:22,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:22,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:22,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:22,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:22,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:22,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:22,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:22,238 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:22,238 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:22,238 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:22,238 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:22,238 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:22,238 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:22,241 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:22,241 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:22,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:22,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:22,241 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:22,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:22,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:22,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:22,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:22,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:22,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:22,242 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:22,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:22,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:22,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:22,243 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:22,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:22,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,245 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,245 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:22,245 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:22,245 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,245 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:22,245 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:22,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:22,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:22,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:22,245 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:22,245 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:22,245 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:22,245 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:56:22,245 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:22,245 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:22,245 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:22,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:22,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:22,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:22,296 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:22,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60448 deadline: 1681034242295, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,300 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:22,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60436 deadline: 1681034242298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,303 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:22,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60460 deadline: 1681034242300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,305 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:22,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60506 deadline: 1681034242302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,321 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:22,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60498 deadline: 1681034242319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,337 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=684 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/15f73fd5309d4086aa4a3d968f65a836 2023-04-09 09:56:22,345 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:22,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:22,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:22,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:22,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:22,356 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:22,356 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:22,356 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:22,356 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:22,357 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:22,357 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:22,358 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,358 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:22,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:22,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:22,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:22,358 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,359 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:22,359 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:22,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:22,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:22,359 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:22,368 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:22,368 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:22,368 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:22,368 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:22,368 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:22,368 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:22,368 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:22,368 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:22,368 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:22,368 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:22,368 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:22,368 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:22,368 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:22,368 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,368 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:22,368 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:22,370 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,370 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:22,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:22,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:22,370 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:22,370 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:22,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:22,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:22,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:22,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:22,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:22,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c6a7785[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:22,371 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:22,371 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:22,376 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:22,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:22,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:22,376 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:22,376 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:22,377 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:22,377 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,377 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:22,378 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,378 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:22,378 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:22,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:22,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:22,378 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:22,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:22,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:22,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:22,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:22,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:22,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:22,380 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:22,380 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:22,380 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:22,380 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:22,380 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:22,380 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:22,381 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:22,381 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:22,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:22,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:22,381 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:22,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:22,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:22,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:22,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:22,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:22,382 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:22,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:22,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:22,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:22,383 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:22,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:22,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:22,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,385 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,385 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:22,385 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:22,385 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:22,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:22,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:22,385 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,385 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:22,385 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:22,385 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:56:22,385 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:22,385 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:22,385 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:22,385 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:22,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:22,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:22,385 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:22,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:22,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:22,485 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:22,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:22,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:22,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:22,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:22,489 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:22,489 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:22,489 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:22,489 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:22,489 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:22,490 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:22,492 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:22,492 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:22,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:22,492 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,492 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:22,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:22,493 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:22,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:22,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:22,494 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:22,495 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:22,495 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:22,495 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:22,495 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:22,495 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:22,495 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:22,495 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:22,495 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:22,495 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:22,495 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:22,495 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:22,495 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:22,495 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:22,495 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,496 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:22,496 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:22,497 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,497 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:22,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:22,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:22,498 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:22,498 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:22,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:22,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:22,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:22,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:22,499 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:22,499 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69f323a1[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:22,500 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:22,500 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:22,501 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:22,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:22,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:22,501 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:22,501 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:22,501 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:22,502 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,502 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:22,503 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:22,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:22,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:22,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:22,504 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:22,505 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:22,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:22,505 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:22,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:22,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:22,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:22,507 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:22,507 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:22,507 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:22,507 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:22,507 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:22,507 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:22,508 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:22,508 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:22,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:22,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:22,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:22,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:22,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:22,508 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:22,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:22,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:22,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:22,509 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:22,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:22,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:22,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:22,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,511 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:22,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:22,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,515 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:22,516 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:22,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:22,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:22,516 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:22,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:22,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:22,516 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,516 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:22,516 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:22,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:22,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:22,516 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:22,516 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:22,516 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,516 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:22,517 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:22,517 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:22,616 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:22,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:22,619 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:22,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:22,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:22,620 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:22,620 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:22,620 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:22,621 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:22,621 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:22,621 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:22,622 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,622 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:22,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:22,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:22,623 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,623 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:22,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:22,624 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:22,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:22,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:22,624 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:22,625 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:22,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60498 deadline: 1681034242624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,626 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:22,627 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:22,627 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:22,627 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:22,627 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:22,627 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:22,627 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:22,627 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:22,627 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:22,627 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:22,627 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:22,627 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:22,627 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:22,627 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,628 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:22,628 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:22,629 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,629 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:22,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:22,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:22,630 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:22,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:22,630 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:22,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:22,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:22,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:22,632 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:22,632 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42698e84[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:22,632 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:22,632 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:22,633 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:22,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:22,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:22,633 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:22,633 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:22,633 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:22,633 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,633 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:22,635 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,635 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:22,635 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:22,635 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:22,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:22,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:22,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:22,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:22,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:22,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:22,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:22,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:22,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:22,639 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:22,639 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:22,639 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:22,639 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:22,639 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:22,643 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:22,643 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:22,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:22,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:22,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:22,643 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:22,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:22,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:22,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:22,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:22,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:22,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:22,644 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:22,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:22,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:22,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:22,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:22,647 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,651 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:22,651 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,651 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:22,651 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:22,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:22,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:22,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:22,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:22,651 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:22,651 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:22,651 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:22,651 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,651 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:22,651 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:22,652 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:22,652 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:22,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:22,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:22,752 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:22,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:22,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:22,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:22,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:22,755 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:22,755 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:22,755 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:22,755 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=684 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/C/ff563ed316b747ed9fe2f0d6fbc37028 2023-04-09 09:56:22,755 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:22,757 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:22,757 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:22,759 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:22,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:22,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:22,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:22,760 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,760 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:22,761 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,761 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:22,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:22,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:22,762 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:22,763 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:22,763 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:22,763 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:22,763 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:22,763 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:22,763 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:22,763 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:22,763 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:22,763 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:22,763 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:22,763 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:22,763 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:22,763 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:22,763 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,764 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:22,764 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:22,764 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/A/e932dcb703b3404aaf865d9d31eb8e1a as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/e932dcb703b3404aaf865d9d31eb8e1a 2023-04-09 09:56:22,766 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,766 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:22,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:22,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:22,766 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:22,766 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:22,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:22,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:22,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:22,768 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/e932dcb703b3404aaf865d9d31eb8e1a, entries=350, sequenceid=684, filesize=52.9 K 2023-04-09 09:56:22,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:22,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:22,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e08347a[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:22,769 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/15f73fd5309d4086aa4a3d968f65a836 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/15f73fd5309d4086aa4a3d968f65a836 2023-04-09 09:56:22,769 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:22,769 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:22,770 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:22,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:22,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:22,771 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:22,771 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:22,771 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:22,771 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,771 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:22,773 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/15f73fd5309d4086aa4a3d968f65a836, entries=150, sequenceid=684, filesize=11.9 K 2023-04-09 09:56:22,774 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/C/ff563ed316b747ed9fe2f0d6fbc37028 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/ff563ed316b747ed9fe2f0d6fbc37028 2023-04-09 09:56:22,775 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,775 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:22,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:22,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:22,776 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:22,776 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:22,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:22,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:22,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:22,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:22,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:22,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,781 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/ff563ed316b747ed9fe2f0d6fbc37028, entries=150, sequenceid=684, filesize=11.9 K 2023-04-09 09:56:22,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:22,781 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:22,781 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:22,781 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:22,781 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:22,781 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:22,781 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:22,782 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for c8127616cf5355dd7f1db99764192b95 in 896ms, sequenceid=684, compaction requested=true 2023-04-09 09:56:22,782 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c8127616cf5355dd7f1db99764192b95: 2023-04-09 09:56:22,782 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:56:22,782 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-09 09:56:22,782 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-09 09:56:22,782 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:56:22,782 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:56:22,783 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:22,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:22,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:22,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:22,783 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 107396 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-09 09:56:22,783 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37682 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-09 09:56:22,783 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): c8127616cf5355dd7f1db99764192b95/B is initiating minor compaction (all files) 2023-04-09 09:56:22,783 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c8127616cf5355dd7f1db99764192b95/B in TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:22,783 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/733f3b3ea567457c96f0d4f49461378c, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/36b8434a72f24a188893c8738e997c94, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/15f73fd5309d4086aa4a3d968f65a836] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp, totalSize=36.8 K 2023-04-09 09:56:22,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:22,783 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:22,783 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1912): c8127616cf5355dd7f1db99764192b95/A is initiating minor compaction (all files) 2023-04-09 09:56:22,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:22,784 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c8127616cf5355dd7f1db99764192b95/A in TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:22,784 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:22,784 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/4ea33ebfce6b49ec88127a20a3322cb0, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/72f6462c665845c0b3e5fe2be869c43c, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/e932dcb703b3404aaf865d9d31eb8e1a] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp, totalSize=104.9 K 2023-04-09 09:56:22,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:22,784 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 733f3b3ea567457c96f0d4f49461378c, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=651, earliestPutTs=1681034180170 2023-04-09 09:56:22,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:22,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:22,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:22,784 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 36b8434a72f24a188893c8738e997c94, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=673, earliestPutTs=1681034180729 2023-04-09 09:56:22,784 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 4ea33ebfce6b49ec88127a20a3322cb0, keycount=150, bloomtype=ROW, size=26.5 K, encoding=NONE, compression=NONE, seqNum=651, earliestPutTs=1681034180170 2023-04-09 09:56:22,785 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 15f73fd5309d4086aa4a3d968f65a836, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=684, earliestPutTs=1681034181879 2023-04-09 09:56:22,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:22,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:22,785 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting 72f6462c665845c0b3e5fe2be869c43c, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=673, earliestPutTs=1681034180729 2023-04-09 09:56:22,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:22,786 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] compactions.Compactor(207): Compacting e932dcb703b3404aaf865d9d31eb8e1a, keycount=350, bloomtype=ROW, size=52.9 K, encoding=NONE, compression=NONE, seqNum=684, earliestPutTs=1681034181868 2023-04-09 09:56:22,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:22,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:22,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,788 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,793 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,799 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): c8127616cf5355dd7f1db99764192b95#B#compaction#660 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:56:22,799 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:22,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:22,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:22,799 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:22,799 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:22,799 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:22,799 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:22,799 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:22,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:22,799 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:22,800 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:22,800 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,800 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:22,800 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:22,800 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:22,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:22,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:22,821 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] throttle.PressureAwareThroughputController(145): c8127616cf5355dd7f1db99764192b95#A#compaction#661 average throughput is 0.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:56:22,839 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/b24bb288dfc3453abc3b1f8b3ac15fe8 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/b24bb288dfc3453abc3b1f8b3ac15fe8 2023-04-09 09:56:22,844 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c8127616cf5355dd7f1db99764192b95/B of c8127616cf5355dd7f1db99764192b95 into b24bb288dfc3453abc3b1f8b3ac15fe8(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:56:22,844 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c8127616cf5355dd7f1db99764192b95: 2023-04-09 09:56:22,844 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95., storeName=c8127616cf5355dd7f1db99764192b95/B, priority=13, startTime=1681034182782; duration=0sec 2023-04-09 09:56:22,844 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:56:22,844 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-09 09:56:22,845 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37614 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-09 09:56:22,846 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1912): c8127616cf5355dd7f1db99764192b95/C is initiating minor compaction (all files) 2023-04-09 09:56:22,846 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c8127616cf5355dd7f1db99764192b95/C in TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:22,846 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/b746c6c37ca249979c59d4f35a07b458, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/8f5a2dce2ec74cb880aedf8b2feef72b, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/ff563ed316b747ed9fe2f0d6fbc37028] into tmpdir=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp, totalSize=36.7 K 2023-04-09 09:56:22,846 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting b746c6c37ca249979c59d4f35a07b458, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=651, earliestPutTs=1681034180170 2023-04-09 09:56:22,846 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting 8f5a2dce2ec74cb880aedf8b2feef72b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=673, earliestPutTs=1681034180729 2023-04-09 09:56:22,847 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] compactions.Compactor(207): Compacting ff563ed316b747ed9fe2f0d6fbc37028, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=684, earliestPutTs=1681034181879 2023-04-09 09:56:22,865 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] throttle.PressureAwareThroughputController(145): c8127616cf5355dd7f1db99764192b95#C#compaction#662 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-09 09:56:22,895 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/C/19e943f047004a5b8506e3e0f286c913 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/19e943f047004a5b8506e3e0f286c913 2023-04-09 09:56:22,899 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:22,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:22,901 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c8127616cf5355dd7f1db99764192b95/C of c8127616cf5355dd7f1db99764192b95 into 19e943f047004a5b8506e3e0f286c913(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:56:22,902 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c8127616cf5355dd7f1db99764192b95: 2023-04-09 09:56:22,902 INFO [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95., storeName=c8127616cf5355dd7f1db99764192b95/C, priority=13, startTime=1681034182782; duration=0sec 2023-04-09 09:56:22,902 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:56:22,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:22,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:22,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:22,905 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:22,905 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:22,905 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:22,905 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:22,905 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:22,905 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:22,907 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:22,907 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:22,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:22,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:22,907 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:22,907 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:22,908 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:22,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:22,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:22,908 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:22,911 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/A/74641726bb6748d7a697ff4a15ad44f4 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/74641726bb6748d7a697ff4a15ad44f4 2023-04-09 09:56:22,915 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c8127616cf5355dd7f1db99764192b95/A of c8127616cf5355dd7f1db99764192b95 into 74641726bb6748d7a697ff4a15ad44f4(size=26.6 K), total size for store is 26.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-09 09:56:22,915 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c8127616cf5355dd7f1db99764192b95: 2023-04-09 09:56:22,915 INFO [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95., storeName=c8127616cf5355dd7f1db99764192b95/A, priority=13, startTime=1681034182782; duration=0sec 2023-04-09 09:56:22,915 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:56:22,916 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:22,916 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:22,916 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:22,916 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:22,916 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:22,916 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:22,916 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:22,916 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:22,916 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:22,917 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c8127616cf5355dd7f1db99764192b95 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-09 09:56:22,917 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=A 2023-04-09 09:56:22,917 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:56:22,917 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=B 2023-04-09 09:56:22,917 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:56:22,917 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=C 2023-04-09 09:56:22,917 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:56:23,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on c8127616cf5355dd7f1db99764192b95 2023-04-09 09:56:23,130 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:23,147 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:23,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60498 deadline: 1681034243145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:23,251 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:23,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60498 deadline: 1681034243249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:23,332 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230409fef7e78651dd4a70af95af6b9f4963bb to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409fef7e78651dd4a70af95af6b9f4963bb 2023-04-09 09:56:23,455 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:23,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60498 deadline: 1681034243453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:23,738 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=725, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/A/9be67c50a646452c88cfd8a7cd051680 2023-04-09 09:56:23,751 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=725 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/68153dba05a247df92e07774383d0c38 2023-04-09 09:56:23,759 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:23,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60498 deadline: 1681034243756, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:23,761 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=725 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/C/4c855bad02974f2fac9c964e59566c9f 2023-04-09 09:56:23,767 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/A/9be67c50a646452c88cfd8a7cd051680 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/9be67c50a646452c88cfd8a7cd051680 2023-04-09 09:56:23,770 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/9be67c50a646452c88cfd8a7cd051680, entries=150, sequenceid=725, filesize=25.4 K 2023-04-09 09:56:23,770 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/68153dba05a247df92e07774383d0c38 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/68153dba05a247df92e07774383d0c38 2023-04-09 09:56:23,775 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/68153dba05a247df92e07774383d0c38, entries=150, sequenceid=725, filesize=11.9 K 2023-04-09 09:56:23,775 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/C/4c855bad02974f2fac9c964e59566c9f as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/4c855bad02974f2fac9c964e59566c9f 2023-04-09 09:56:23,778 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/4c855bad02974f2fac9c964e59566c9f, entries=150, sequenceid=725, filesize=11.9 K 2023-04-09 09:56:23,779 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for c8127616cf5355dd7f1db99764192b95 in 862ms, sequenceid=725, compaction requested=false 2023-04-09 09:56:23,779 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c8127616cf5355dd7f1db99764192b95: 2023-04-09 09:56:23,779 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:23,779 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:23,779 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:23,779 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:23,779 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:23,779 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:23,781 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:23,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:23,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:23,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:23,781 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:23,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:23,782 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:23,782 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:23,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:23,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:23,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:23,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:23,784 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:23,784 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2131b461[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:23,784 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:23,784 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:23,785 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:23,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:23,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:23,785 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:23,785 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:23,785 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:23,785 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:23,786 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:23,786 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:23,786 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:23,786 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:23,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:23,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:23,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:23,787 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:23,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:23,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:23,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:23,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:23,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:23,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:23,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:23,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:23,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:23,791 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:23,791 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:23,791 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:23,791 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:23,791 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:23,793 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:23,793 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:23,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:23,793 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:23,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:23,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:23,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:23,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:23,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:23,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:23,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:23,794 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:23,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:23,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:23,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:23,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:23,795 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:23,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:23,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:23,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:23,798 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:23,798 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:23,798 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:23,799 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:23,799 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:23,799 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:23,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:23,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:23,799 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:23,799 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:23,799 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:56:23,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:23,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:23,799 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:23,799 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:23,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:23,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:23,799 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:23,799 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:23,899 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:23,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:23,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:23,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:23,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:23,903 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:23,903 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:23,903 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:23,903 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:23,903 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:23,903 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:23,904 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:23,904 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:23,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:23,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:23,904 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:23,904 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:23,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:23,905 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:23,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:23,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:23,905 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:23,905 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:23,905 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:23,905 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:23,905 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:23,906 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:23,906 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:23,906 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:23,906 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:23,906 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:23,906 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c8127616cf5355dd7f1db99764192b95 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-09 09:56:23,906 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=A 2023-04-09 09:56:23,906 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:56:23,906 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=B 2023-04-09 09:56:23,907 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:56:23,907 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=C 2023-04-09 09:56:23,907 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:56:24,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on c8127616cf5355dd7f1db99764192b95 2023-04-09 09:56:24,262 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:24,318 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304090228e87b0c82426aa6ddf8afdbf5f218 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304090228e87b0c82426aa6ddf8afdbf5f218 2023-04-09 09:56:24,319 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681034184268, max timestamp=1681034184318Num uniques 859; 2023-04-09 09:56:24,319 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681034184268, max timestamp=1681034184318Num uniques 859; 2023-04-09 09:56:24,320 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-09 09:56:24,320 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-09 09:56:24,321 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681034184268, max timestamp=1681034184318Num uniques 859; 2023-04-09 09:56:24,321 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-09 09:56:24,368 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:24,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60436 deadline: 1681034244362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:24,368 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:24,368 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:24,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60448 deadline: 1681034244363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:24,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60460 deadline: 1681034244362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:24,369 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:24,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60506 deadline: 1681034244366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:24,372 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:24,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60498 deadline: 1681034244368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:24,470 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:24,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60436 deadline: 1681034244469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:24,471 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:24,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60448 deadline: 1681034244469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:24,471 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:24,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60460 deadline: 1681034244470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:24,472 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:24,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60506 deadline: 1681034244470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:24,474 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:24,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60498 deadline: 1681034244473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:24,674 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:24,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60436 deadline: 1681034244672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:24,674 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:24,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60460 deadline: 1681034244672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:24,674 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:24,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60448 deadline: 1681034244673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:24,675 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:24,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60506 deadline: 1681034244673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:24,678 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:24,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60498 deadline: 1681034244676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:24,736 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=735, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/A/4decbb2e31ff4056b34a1463a88a5775 2023-04-09 09:56:24,747 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=735 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/b8c419a2881b46d4917ab7d87edc86cb 2023-04-09 09:56:24,977 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:24,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60436 deadline: 1681034244975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:24,978 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:24,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60506 deadline: 1681034244976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:24,980 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:24,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60448 deadline: 1681034244976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:24,980 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:24,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60460 deadline: 1681034244977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:24,984 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:24,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60498 deadline: 1681034244981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:25,159 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=735 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/C/b39e5ad44af349deb6d9a81773615b58 2023-04-09 09:56:25,163 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/A/4decbb2e31ff4056b34a1463a88a5775 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/4decbb2e31ff4056b34a1463a88a5775 2023-04-09 09:56:25,169 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/4decbb2e31ff4056b34a1463a88a5775, entries=150, sequenceid=735, filesize=25.4 K 2023-04-09 09:56:25,170 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/b8c419a2881b46d4917ab7d87edc86cb as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/b8c419a2881b46d4917ab7d87edc86cb 2023-04-09 09:56:25,174 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/b8c419a2881b46d4917ab7d87edc86cb, entries=150, sequenceid=735, filesize=11.9 K 2023-04-09 09:56:25,175 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/C/b39e5ad44af349deb6d9a81773615b58 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/b39e5ad44af349deb6d9a81773615b58 2023-04-09 09:56:25,181 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/b39e5ad44af349deb6d9a81773615b58, entries=150, sequenceid=735, filesize=11.9 K 2023-04-09 09:56:25,183 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=138.65 KB/141980 for c8127616cf5355dd7f1db99764192b95 in 1277ms, sequenceid=735, compaction requested=true 2023-04-09 09:56:25,183 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c8127616cf5355dd7f1db99764192b95: 2023-04-09 09:56:25,183 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:25,183 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:25,183 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:25,184 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:25,186 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:25,187 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:25,188 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:25,188 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:25,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:25,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:25,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:25,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:25,189 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:25,190 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:25,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:25,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:25,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:25,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:25,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:25,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@512f65cd[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:25,191 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:25,191 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:25,193 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:25,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:25,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:25,194 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:25,194 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:25,194 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:25,194 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:25,194 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:25,195 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:25,195 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:25,195 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:25,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:25,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:25,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:25,195 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:25,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:25,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:25,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:25,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:25,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:25,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:25,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:25,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:25,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:25,198 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:25,198 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:25,198 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:25,198 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:25,198 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:25,200 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:25,200 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:25,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:25,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:25,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:25,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:25,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:25,200 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:25,200 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:25,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:25,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:25,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:25,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:25,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:25,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:25,202 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:25,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:25,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:25,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:25,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:25,204 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:25,204 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:25,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:25,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:25,204 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:25,204 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:25,205 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:25,205 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:25,205 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:25,205 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:25,205 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:25,205 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:56:25,205 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:25,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:25,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:25,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:25,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:25,205 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:25,205 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:25,305 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:25,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:25,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-09 09:56:25,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-09 09:56:25,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-09 09:56:25,317 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-09 09:56:25,317 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:25,318 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-09 09:56:25,318 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-09 09:56:25,318 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:25,318 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:25,320 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:25,320 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:25,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:25,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:25,320 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:25,320 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-09 09:56:25,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:25,321 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:25,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-09 09:56:25,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:25,321 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-09 09:56:25,322 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-09 09:56:25,322 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-09 09:56:25,322 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-09 09:56:25,322 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-09 09:56:25,322 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-09 09:56:25,322 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-09 09:56:25,322 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-09 09:56:25,322 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:25,322 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. started... 2023-04-09 09:56:25,323 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c8127616cf5355dd7f1db99764192b95 3/3 column families, dataSize=138.65 KB heapSize=395.14 KB 2023-04-09 09:56:25,323 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=A 2023-04-09 09:56:25,323 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:56:25,323 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=B 2023-04-09 09:56:25,323 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:56:25,323 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=C 2023-04-09 09:56:25,323 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-09 09:56:25,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on c8127616cf5355dd7f1db99764192b95 2023-04-09 09:56:25,482 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. as already flushing 2023-04-09 09:56:25,490 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:25,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60460 deadline: 1681034245487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:25,491 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:25,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60506 deadline: 1681034245487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:25,491 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:25,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 287 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60498 deadline: 1681034245487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:25,493 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:25,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60448 deadline: 1681034245489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:25,493 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:25,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60436 deadline: 1681034245490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:25,594 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:25,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60460 deadline: 1681034245592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:25,594 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:25,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60506 deadline: 1681034245592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:25,598 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:25,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60448 deadline: 1681034245594, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:25,598 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:25,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42467] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60436 deadline: 1681034245595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:25,633 DEBUG [Thread-4063] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x25baee3e to 127.0.0.1:57816 2023-04-09 09:56:25,633 DEBUG [Thread-4063] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-09 09:56:25,634 DEBUG [Thread-4065] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2c53bf96 to 127.0.0.1:57816 2023-04-09 09:56:25,634 DEBUG [Thread-4065] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-09 09:56:25,635 DEBUG [Thread-4067] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3b59bdcf to 127.0.0.1:57816 2023-04-09 09:56:25,635 DEBUG [Thread-4067] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-09 09:56:25,635 DEBUG [Thread-4069] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3d765193 to 127.0.0.1:57816 2023-04-09 09:56:25,635 DEBUG [Thread-4069] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-09 09:56:25,637 DEBUG [Thread-4071] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x32bf16cd to 127.0.0.1:57816 2023-04-09 09:56:25,637 DEBUG [Thread-4071] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-09 09:56:25,737 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230409812a8b1854d84c5eb889f66ea1603e32 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409812a8b1854d84c5eb889f66ea1603e32 2023-04-09 09:56:25,795 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:25,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60506 deadline: 1681034245795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:25,796 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:25,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60460 deadline: 1681034245796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:25,799 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:25,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42467] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60448 deadline: 1681034245799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:25,799 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:25,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60436 deadline: 1681034245799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:26,096 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:26,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60506 deadline: 1681034246096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:26,099 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:26,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60460 deadline: 1681034246099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:26,101 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:26,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60436 deadline: 1681034246101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:26,102 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:26,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60448 deadline: 1681034246102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:26,145 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=772, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/A/dcd27f8800664591bdd5b262715d7379 2023-04-09 09:56:26,496 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:26,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 289 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60498 deadline: 1681034246496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:26,554 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=772 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/c1b6b91c2ea94a30834cd652ca578b82 2023-04-09 09:56:26,600 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:26,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 294 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60506 deadline: 1681034246600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:26,601 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:26,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60460 deadline: 1681034246600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:26,602 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:26,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60448 deadline: 1681034246602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:26,604 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-09 09:56:26,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60436 deadline: 1681034246604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:26,962 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=772 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/C/54e2d00e052d403db49d420f23c36873 2023-04-09 09:56:26,965 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/A/dcd27f8800664591bdd5b262715d7379 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/dcd27f8800664591bdd5b262715d7379 2023-04-09 09:56:26,968 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/dcd27f8800664591bdd5b262715d7379, entries=150, sequenceid=772, filesize=25.4 K 2023-04-09 09:56:26,968 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/c1b6b91c2ea94a30834cd652ca578b82 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/c1b6b91c2ea94a30834cd652ca578b82 2023-04-09 09:56:26,971 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/c1b6b91c2ea94a30834cd652ca578b82, entries=150, sequenceid=772, filesize=11.9 K 2023-04-09 09:56:26,972 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/C/54e2d00e052d403db49d420f23c36873 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/54e2d00e052d403db49d420f23c36873 2023-04-09 09:56:26,976 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/54e2d00e052d403db49d420f23c36873, entries=150, sequenceid=772, filesize=11.9 K 2023-04-09 09:56:26,976 INFO [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~138.65 KB/141980, heapSize ~395.09 KB/404576, currentSize=40.25 KB/41220 for c8127616cf5355dd7f1db99764192b95 in 1653ms, sequenceid=772, compaction requested=true 2023-04-09 09:56:26,976 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c8127616cf5355dd7f1db99764192b95: 2023-04-09 09:56:26,976 DEBUG [rs(jenkins-hbase4.apache.org,42467,1681034019544)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:26,976 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-09 09:56:26,977 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-09 09:56:26,977 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:26,977 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-09 09:56:26,977 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-09 09:56:26,978 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:26,978 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:26,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:26,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:26,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:26,978 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:26,978 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-09 09:56:26,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:26,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:26,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:26,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:26,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:26,980 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42467,1681034019544' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-09 09:56:26,980 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5fbcdcc2[Count = 0] remaining members to acquire global barrier 2023-04-09 09:56:26,980 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-09 09:56:26,980 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:26,981 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:26,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:26,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:26,981 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-09 09:56:26,981 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:26,981 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-09 09:56:26,981 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-09 09:56:26,981 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42467,1681034019544' in zk 2023-04-09 09:56:26,985 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:26,985 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-09 09:56:26,985 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:26,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:26,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:26,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:26,985 DEBUG [member: 'jenkins-hbase4.apache.org,42467,1681034019544' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-09 09:56:26,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:26,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:26,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:26,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:26,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:26,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:26,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:26,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42467,1681034019544': 2023-04-09 09:56:26,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42467,1681034019544' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-09 09:56:26,989 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-09 09:56:26,989 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-09 09:56:26,989 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-09 09:56:26,989 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-09 09:56:26,989 INFO [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-09 09:56:26,990 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:26,990 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:26,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:26,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-09 09:56:26,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-09 09:56:26,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:26,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:26,990 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:26,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-09 09:56:26,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:26,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:26,991 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:26,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:26,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:26,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-09 09:56:26,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:26,992 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:26,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:26,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-09 09:56:26,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-09 09:56:26,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:26,994 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:26,994 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:26,994 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-09 09:56:26,994 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-09 09:56:26,994 DEBUG [(jenkins-hbase4.apache.org,41863,1681034018311)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-09 09:56:26,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-09 09:56:26,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-09 09:56:26,994 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:26,994 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-09 09:56:26,994 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-09 09:56:26,994 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-09 09:56:26,995 DEBUG [Thread-4062] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-09 09:56:26,995 DEBUG [Thread-4062] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-09 09:56:26,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-09 09:56:26,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-09 09:56:26,994 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:26,995 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-09 09:56:26,995 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-09 09:56:26,996 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-09 09:56:27,095 DEBUG [Thread-4062] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-09 09:56:27,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-09 09:56:27,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42467] regionserver.HRegion(9158): Flush requested on c8127616cf5355dd7f1db99764192b95 2023-04-09 09:56:27,603 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c8127616cf5355dd7f1db99764192b95 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-09 09:56:27,603 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=A 2023-04-09 09:56:27,603 DEBUG [Thread-4060] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x08b57aae to 127.0.0.1:57816 2023-04-09 09:56:27,603 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:56:27,603 DEBUG [Thread-4060] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-09 09:56:27,603 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=B 2023-04-09 09:56:27,603 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:56:27,603 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=C 2023-04-09 09:56:27,603 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:56:27,612 DEBUG [Thread-4054] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6f06dd50 to 127.0.0.1:57816 2023-04-09 09:56:27,612 DEBUG [Thread-4054] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-09 09:56:27,612 DEBUG [Thread-4052] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1e3503b4 to 127.0.0.1:57816 2023-04-09 09:56:27,612 DEBUG [Thread-4052] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-09 09:56:27,612 DEBUG [Thread-4058] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4db22237 to 127.0.0.1:57816 2023-04-09 09:56:27,612 DEBUG [Thread-4058] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-09 09:56:28,020 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230409bbee2b1833634b2e92ec5b6083b8b666 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409bbee2b1833634b2e92ec5b6083b8b666 2023-04-09 09:56:28,424 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=782, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/A/13c9425c5df4470c9d1e90b0c786ff5b 2023-04-09 09:56:28,431 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=782 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/fe7f706c60c8462884b34e4a7101d553 2023-04-09 09:56:28,500 DEBUG [Thread-4056] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5f9e8654 to 127.0.0.1:57816 2023-04-09 09:56:28,500 DEBUG [Thread-4056] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-09 09:56:28,500 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-09 09:56:28,501 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(394): wrote 119 2023-04-09 09:56:28,501 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(394): wrote 161 2023-04-09 09:56:28,501 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(394): wrote 168 2023-04-09 09:56:28,501 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(394): wrote 127 2023-04-09 09:56:28,501 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(394): wrote 84 2023-04-09 09:56:28,501 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-09 09:56:28,501 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(398): read 4195 2023-04-09 09:56:28,501 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(398): read 4210 2023-04-09 09:56:28,501 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(398): read 4198 2023-04-09 09:56:28,501 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(398): read 4223 2023-04-09 09:56:28,501 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(398): read 4202 2023-04-09 09:56:28,501 INFO [Listener at localhost/44563] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-09 09:56:28,501 INFO [Listener at localhost/44563] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-09 09:56:28,501 DEBUG [Listener at localhost/44563] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x470742aa to 127.0.0.1:57816 2023-04-09 09:56:28,501 DEBUG [Listener at localhost/44563] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-09 09:56:28,502 INFO [Listener at localhost/44563] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-09 09:56:28,502 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-09 09:56:28,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-09 09:56:28,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-09 09:56:28,505 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681034188505"}]},"ts":"1681034188505"} 2023-04-09 09:56:28,506 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-09 09:56:28,507 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-09 09:56:28,507 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c8127616cf5355dd7f1db99764192b95, UNASSIGN}] 2023-04-09 09:56:28,509 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c8127616cf5355dd7f1db99764192b95, UNASSIGN 2023-04-09 09:56:28,509 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=c8127616cf5355dd7f1db99764192b95, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:28,509 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681034188509"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681034188509"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681034188509"}]},"ts":"1681034188509"} 2023-04-09 09:56:28,510 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE; CloseRegionProcedure c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544}] 2023-04-09 09:56:28,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-09 09:56:28,661 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close c8127616cf5355dd7f1db99764192b95 2023-04-09 09:56:28,661 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing c8127616cf5355dd7f1db99764192b95, disabling compactions & flushes 2023-04-09 09:56:28,661 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:28,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-09 09:56:28,838 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=782 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/C/ac5a6cf4169f4d4595746db36350c98c 2023-04-09 09:56:28,841 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/A/13c9425c5df4470c9d1e90b0c786ff5b as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/13c9425c5df4470c9d1e90b0c786ff5b 2023-04-09 09:56:28,844 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/13c9425c5df4470c9d1e90b0c786ff5b, entries=150, sequenceid=782, filesize=25.4 K 2023-04-09 09:56:28,845 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/fe7f706c60c8462884b34e4a7101d553 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/fe7f706c60c8462884b34e4a7101d553 2023-04-09 09:56:28,848 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/fe7f706c60c8462884b34e4a7101d553, entries=150, sequenceid=782, filesize=11.9 K 2023-04-09 09:56:28,849 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/C/ac5a6cf4169f4d4595746db36350c98c as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/ac5a6cf4169f4d4595746db36350c98c 2023-04-09 09:56:28,852 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/ac5a6cf4169f4d4595746db36350c98c, entries=150, sequenceid=782, filesize=11.9 K 2023-04-09 09:56:28,853 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=26.84 KB/27480 for c8127616cf5355dd7f1db99764192b95 in 1250ms, sequenceid=782, compaction requested=true 2023-04-09 09:56:28,853 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c8127616cf5355dd7f1db99764192b95: 2023-04-09 09:56:28,853 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:28,853 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:56:28,853 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. because compaction request was cancelled 2023-04-09 09:56:28,853 DEBUG [RS:0;jenkins-hbase4:42467-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. because compaction request was cancelled 2023-04-09 09:56:28,853 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-09 09:56:28,853 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:28,853 DEBUG [RS:0;jenkins-hbase4:42467-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. because compaction request was cancelled 2023-04-09 09:56:28,853 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. after waiting 0 ms 2023-04-09 09:56:28,853 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:28,853 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-09 09:56:28,853 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing c8127616cf5355dd7f1db99764192b95 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-09 09:56:28,853 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=A 2023-04-09 09:56:28,853 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:56:28,854 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=B 2023-04-09 09:56:28,854 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:56:28,854 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c8127616cf5355dd7f1db99764192b95, store=C 2023-04-09 09:56:28,854 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-09 09:56:29,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-09 09:56:29,266 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040976d6cdc2873d4c2bba922b6d52d87d07 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040976d6cdc2873d4c2bba922b6d52d87d07 2023-04-09 09:56:29,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-09 09:56:29,670 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=789, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/A/249d084fc2bb4ab3b97ae45d6c6ca520 2023-04-09 09:56:30,078 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=789 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/70e66992b9734d4181587653dfc9ad6e 2023-04-09 09:56:30,486 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=789 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/C/6a7f2e15a74d4864830c4623d612379a 2023-04-09 09:56:30,489 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/A/249d084fc2bb4ab3b97ae45d6c6ca520 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/249d084fc2bb4ab3b97ae45d6c6ca520 2023-04-09 09:56:30,492 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/249d084fc2bb4ab3b97ae45d6c6ca520, entries=150, sequenceid=789, filesize=25.4 K 2023-04-09 09:56:30,493 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/B/70e66992b9734d4181587653dfc9ad6e as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/70e66992b9734d4181587653dfc9ad6e 2023-04-09 09:56:30,496 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/70e66992b9734d4181587653dfc9ad6e, entries=150, sequenceid=789, filesize=11.9 K 2023-04-09 09:56:30,497 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/.tmp/C/6a7f2e15a74d4864830c4623d612379a as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/6a7f2e15a74d4864830c4623d612379a 2023-04-09 09:56:30,500 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/6a7f2e15a74d4864830c4623d612379a, entries=150, sequenceid=789, filesize=11.9 K 2023-04-09 09:56:30,501 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for c8127616cf5355dd7f1db99764192b95 in 1648ms, sequenceid=789, compaction requested=true 2023-04-09 09:56:30,538 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/057ab254f6ce439692bb82f8f79f86b2, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/8e0c3f1efbb849f89322be02af6cb4b3, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/f7a366128bf842cdb23569c24cd5d86c, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/f3f6fd2b45e647a5922bd4ea4a561cb4, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/aed1071f46a94f9b82eacbf3a28f23fc, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/d3891a5d69534554b00b403ba504d73f, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/d795c2cedb8943afaad609b09ea6dfcb, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/033c11eb33fa4911a6ab53bc55dbac9d, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/38eaff8221ad4f91a144403286f0080a, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/e25930902a7346a5a1ac05688b0fc3ae, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/22c905efc1c744049ad4d0d8cb342618, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/19f6883ee25e49caaea691ed2bc7339d, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/0b0b29af87db471da9925cf5270b588d, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/160a7f9c11c24ccb820a59715df9d44b, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/3eb010383f974073a3f30742b41a0725, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/a0eadfb2259b42c5a7a39558c75e0bed, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/4483b4a24bde47c19550c50aad380e57, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/6ae9fd5c7b9f442e977e8c77a2eba500, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/16840ed44c3848278ab30b389bdb74a9, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/a2be7e31374d4492bbb2217370ff0db0, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/2009135c156044ce8c3d4a3633067e56, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/89af7261fb1d4960909a8ceabec39ae9, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/9fd43c0a453d40518e12bdb9585d4bb6, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/facafe1ddc5c4257abae997a789723a4, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/df548ec2fb014b64b8969818d4680fde, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/7642c2b4cefe404585320ea97682e5df, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/f2527b9174e74bf99667a59e5f75d544, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/b127ecd481d54190b3f45c859710248a, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/d30d46ac14f24252ad31aebdad140b0d, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/a20c35cbcad54e808fa506cb7a5be7e4, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/0833cffa55de47b59ea6266b04804399, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/58b11c680a3445998a63e07d66d47e2d, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/4ea33ebfce6b49ec88127a20a3322cb0, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/72f6462c665845c0b3e5fe2be869c43c, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/e932dcb703b3404aaf865d9d31eb8e1a] to archive 2023-04-09 09:56:30,540 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-09 09:56:30,543 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/057ab254f6ce439692bb82f8f79f86b2 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/057ab254f6ce439692bb82f8f79f86b2 2023-04-09 09:56:30,545 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/8e0c3f1efbb849f89322be02af6cb4b3 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/8e0c3f1efbb849f89322be02af6cb4b3 2023-04-09 09:56:30,547 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/f7a366128bf842cdb23569c24cd5d86c to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/f7a366128bf842cdb23569c24cd5d86c 2023-04-09 09:56:30,548 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/f3f6fd2b45e647a5922bd4ea4a561cb4 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/f3f6fd2b45e647a5922bd4ea4a561cb4 2023-04-09 09:56:30,549 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/aed1071f46a94f9b82eacbf3a28f23fc to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/aed1071f46a94f9b82eacbf3a28f23fc 2023-04-09 09:56:30,550 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/d3891a5d69534554b00b403ba504d73f to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/d3891a5d69534554b00b403ba504d73f 2023-04-09 09:56:30,551 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/d795c2cedb8943afaad609b09ea6dfcb to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/d795c2cedb8943afaad609b09ea6dfcb 2023-04-09 09:56:30,552 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/033c11eb33fa4911a6ab53bc55dbac9d to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/033c11eb33fa4911a6ab53bc55dbac9d 2023-04-09 09:56:30,552 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/38eaff8221ad4f91a144403286f0080a to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/38eaff8221ad4f91a144403286f0080a 2023-04-09 09:56:30,554 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/e25930902a7346a5a1ac05688b0fc3ae to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/e25930902a7346a5a1ac05688b0fc3ae 2023-04-09 09:56:30,555 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/22c905efc1c744049ad4d0d8cb342618 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/22c905efc1c744049ad4d0d8cb342618 2023-04-09 09:56:30,556 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/19f6883ee25e49caaea691ed2bc7339d to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/19f6883ee25e49caaea691ed2bc7339d 2023-04-09 09:56:30,558 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/0b0b29af87db471da9925cf5270b588d to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/0b0b29af87db471da9925cf5270b588d 2023-04-09 09:56:30,559 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/160a7f9c11c24ccb820a59715df9d44b to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/160a7f9c11c24ccb820a59715df9d44b 2023-04-09 09:56:30,560 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/3eb010383f974073a3f30742b41a0725 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/3eb010383f974073a3f30742b41a0725 2023-04-09 09:56:30,562 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/a0eadfb2259b42c5a7a39558c75e0bed to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/a0eadfb2259b42c5a7a39558c75e0bed 2023-04-09 09:56:30,563 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/4483b4a24bde47c19550c50aad380e57 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/4483b4a24bde47c19550c50aad380e57 2023-04-09 09:56:30,564 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/6ae9fd5c7b9f442e977e8c77a2eba500 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/6ae9fd5c7b9f442e977e8c77a2eba500 2023-04-09 09:56:30,564 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/16840ed44c3848278ab30b389bdb74a9 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/16840ed44c3848278ab30b389bdb74a9 2023-04-09 09:56:30,565 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/a2be7e31374d4492bbb2217370ff0db0 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/a2be7e31374d4492bbb2217370ff0db0 2023-04-09 09:56:30,566 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/2009135c156044ce8c3d4a3633067e56 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/2009135c156044ce8c3d4a3633067e56 2023-04-09 09:56:30,567 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/89af7261fb1d4960909a8ceabec39ae9 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/89af7261fb1d4960909a8ceabec39ae9 2023-04-09 09:56:30,569 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/9fd43c0a453d40518e12bdb9585d4bb6 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/9fd43c0a453d40518e12bdb9585d4bb6 2023-04-09 09:56:30,571 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/facafe1ddc5c4257abae997a789723a4 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/facafe1ddc5c4257abae997a789723a4 2023-04-09 09:56:30,572 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/df548ec2fb014b64b8969818d4680fde to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/df548ec2fb014b64b8969818d4680fde 2023-04-09 09:56:30,573 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/7642c2b4cefe404585320ea97682e5df to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/7642c2b4cefe404585320ea97682e5df 2023-04-09 09:56:30,574 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/f2527b9174e74bf99667a59e5f75d544 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/f2527b9174e74bf99667a59e5f75d544 2023-04-09 09:56:30,576 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/b127ecd481d54190b3f45c859710248a to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/b127ecd481d54190b3f45c859710248a 2023-04-09 09:56:30,577 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/d30d46ac14f24252ad31aebdad140b0d to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/d30d46ac14f24252ad31aebdad140b0d 2023-04-09 09:56:30,578 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/a20c35cbcad54e808fa506cb7a5be7e4 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/a20c35cbcad54e808fa506cb7a5be7e4 2023-04-09 09:56:30,579 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/0833cffa55de47b59ea6266b04804399 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/0833cffa55de47b59ea6266b04804399 2023-04-09 09:56:30,580 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/58b11c680a3445998a63e07d66d47e2d to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/58b11c680a3445998a63e07d66d47e2d 2023-04-09 09:56:30,582 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/4ea33ebfce6b49ec88127a20a3322cb0 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/4ea33ebfce6b49ec88127a20a3322cb0 2023-04-09 09:56:30,584 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/72f6462c665845c0b3e5fe2be869c43c to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/72f6462c665845c0b3e5fe2be869c43c 2023-04-09 09:56:30,586 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/e932dcb703b3404aaf865d9d31eb8e1a to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/e932dcb703b3404aaf865d9d31eb8e1a 2023-04-09 09:56:30,589 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/129c073ee3e84e3cbc8d2a4fafcdc7bd, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/051d433cf1324af5a626244822009bde, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/a763996c14b5440789cadce1a146905f, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/5e41c9fa189642dbade8cc63d4ecf3a0, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/09b07cc2f6b548d4bb5ebfb8e94a74e1, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/5bda5149d98740a18addd7a26ecca890, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/ca62cc483bc247ae9bbbfb85e87a797d, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/d47f960990664f22ab9c70dae55d83c9, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/686c3ad2a7754a5e909b87deed1874ff, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/65e557d0523d4babb81e58d469ef8ed5, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/74721bdf15624468891165c7f5afdb83, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/2fd097b323724390ac79ca324ff52ac4, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/d462410689894f99a970feb1f8f68bb5, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/a2925f0d1a1a4d188f830cd86c0d5ca4, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/5dcb44ad7dc54c7aa2b4cf5c2ef0e4be, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/9b5a960f42be4d21bb122a55040dce8e, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/08a64638841d437c9c1808dbcce13abb, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/7eb5e647f67b4eb4a7e350e687d51b43, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/1bc7772601404601af8caa10342d6c16, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/8891fd4138e64238a0fe3194fdda1d5d, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/e181c99d2b344c8db6dc3e88c6e97ed0, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/bed95e971b124685834f0015e59cf56a, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/4e4d3cb1eb734e0ab002aded175a0808, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/24ccfbb87828407eb69fdc2c91aef52f, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/52928e7397b3411882eb72c8766e5a08, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/021b5da8dca54f2fb004b648849a503a, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/e21325e9850c40e5b6f5834112b35855, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/21a9451b38c449a685d6662b6bce364b, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/b86f3364105b47fdad86f9c1e2616c99, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/f32324d96c0d4c61a905cdb87b12cd69, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/71e2dc7c4e3f4b5b8d86bbcd1794b24a, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/5ed522c8593044e5b66dc2e2ba59b365, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/733f3b3ea567457c96f0d4f49461378c, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/bd80be54b5e64f308b2ae26f755e3bf6, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/36b8434a72f24a188893c8738e997c94, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/15f73fd5309d4086aa4a3d968f65a836] to archive 2023-04-09 09:56:30,590 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-09 09:56:30,592 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/129c073ee3e84e3cbc8d2a4fafcdc7bd to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/129c073ee3e84e3cbc8d2a4fafcdc7bd 2023-04-09 09:56:30,593 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/051d433cf1324af5a626244822009bde to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/051d433cf1324af5a626244822009bde 2023-04-09 09:56:30,595 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/a763996c14b5440789cadce1a146905f to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/a763996c14b5440789cadce1a146905f 2023-04-09 09:56:30,596 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/5e41c9fa189642dbade8cc63d4ecf3a0 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/5e41c9fa189642dbade8cc63d4ecf3a0 2023-04-09 09:56:30,597 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/09b07cc2f6b548d4bb5ebfb8e94a74e1 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/09b07cc2f6b548d4bb5ebfb8e94a74e1 2023-04-09 09:56:30,598 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/5bda5149d98740a18addd7a26ecca890 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/5bda5149d98740a18addd7a26ecca890 2023-04-09 09:56:30,599 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/ca62cc483bc247ae9bbbfb85e87a797d to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/ca62cc483bc247ae9bbbfb85e87a797d 2023-04-09 09:56:30,600 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/d47f960990664f22ab9c70dae55d83c9 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/d47f960990664f22ab9c70dae55d83c9 2023-04-09 09:56:30,605 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/686c3ad2a7754a5e909b87deed1874ff to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/686c3ad2a7754a5e909b87deed1874ff 2023-04-09 09:56:30,606 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/65e557d0523d4babb81e58d469ef8ed5 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/65e557d0523d4babb81e58d469ef8ed5 2023-04-09 09:56:30,607 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/74721bdf15624468891165c7f5afdb83 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/74721bdf15624468891165c7f5afdb83 2023-04-09 09:56:30,608 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/2fd097b323724390ac79ca324ff52ac4 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/2fd097b323724390ac79ca324ff52ac4 2023-04-09 09:56:30,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-09 09:56:30,609 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/d462410689894f99a970feb1f8f68bb5 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/d462410689894f99a970feb1f8f68bb5 2023-04-09 09:56:30,610 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/a2925f0d1a1a4d188f830cd86c0d5ca4 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/a2925f0d1a1a4d188f830cd86c0d5ca4 2023-04-09 09:56:30,611 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/5dcb44ad7dc54c7aa2b4cf5c2ef0e4be to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/5dcb44ad7dc54c7aa2b4cf5c2ef0e4be 2023-04-09 09:56:30,612 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/9b5a960f42be4d21bb122a55040dce8e to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/9b5a960f42be4d21bb122a55040dce8e 2023-04-09 09:56:30,613 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/08a64638841d437c9c1808dbcce13abb to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/08a64638841d437c9c1808dbcce13abb 2023-04-09 09:56:30,614 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/7eb5e647f67b4eb4a7e350e687d51b43 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/7eb5e647f67b4eb4a7e350e687d51b43 2023-04-09 09:56:30,615 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/1bc7772601404601af8caa10342d6c16 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/1bc7772601404601af8caa10342d6c16 2023-04-09 09:56:30,616 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/8891fd4138e64238a0fe3194fdda1d5d to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/8891fd4138e64238a0fe3194fdda1d5d 2023-04-09 09:56:30,617 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/e181c99d2b344c8db6dc3e88c6e97ed0 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/e181c99d2b344c8db6dc3e88c6e97ed0 2023-04-09 09:56:30,618 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/bed95e971b124685834f0015e59cf56a to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/bed95e971b124685834f0015e59cf56a 2023-04-09 09:56:30,619 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/4e4d3cb1eb734e0ab002aded175a0808 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/4e4d3cb1eb734e0ab002aded175a0808 2023-04-09 09:56:30,619 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/24ccfbb87828407eb69fdc2c91aef52f to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/24ccfbb87828407eb69fdc2c91aef52f 2023-04-09 09:56:30,620 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/52928e7397b3411882eb72c8766e5a08 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/52928e7397b3411882eb72c8766e5a08 2023-04-09 09:56:30,621 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/021b5da8dca54f2fb004b648849a503a to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/021b5da8dca54f2fb004b648849a503a 2023-04-09 09:56:30,622 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/e21325e9850c40e5b6f5834112b35855 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/e21325e9850c40e5b6f5834112b35855 2023-04-09 09:56:30,623 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/21a9451b38c449a685d6662b6bce364b to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/21a9451b38c449a685d6662b6bce364b 2023-04-09 09:56:30,624 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/b86f3364105b47fdad86f9c1e2616c99 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/b86f3364105b47fdad86f9c1e2616c99 2023-04-09 09:56:30,625 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/f32324d96c0d4c61a905cdb87b12cd69 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/f32324d96c0d4c61a905cdb87b12cd69 2023-04-09 09:56:30,625 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/71e2dc7c4e3f4b5b8d86bbcd1794b24a to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/71e2dc7c4e3f4b5b8d86bbcd1794b24a 2023-04-09 09:56:30,626 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/5ed522c8593044e5b66dc2e2ba59b365 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/5ed522c8593044e5b66dc2e2ba59b365 2023-04-09 09:56:30,627 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/733f3b3ea567457c96f0d4f49461378c to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/733f3b3ea567457c96f0d4f49461378c 2023-04-09 09:56:30,628 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/bd80be54b5e64f308b2ae26f755e3bf6 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/bd80be54b5e64f308b2ae26f755e3bf6 2023-04-09 09:56:30,629 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/36b8434a72f24a188893c8738e997c94 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/36b8434a72f24a188893c8738e997c94 2023-04-09 09:56:30,630 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/15f73fd5309d4086aa4a3d968f65a836 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/15f73fd5309d4086aa4a3d968f65a836 2023-04-09 09:56:30,633 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/74b917e760f448caa7aad394f8a80885, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/c8b220063d05497ca6ca47ea08124297, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/d88ac10f0b5a4045a38d2e5441f5c8f0, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/9e49d1dbf7ca4ffc87d8cbfaa3cf445e, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/8e798f5fa9b5494a8e5e449294b29eae, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/9bde71d48cd34d158104f08f38f88121, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/297f6d7fced140c9ac8a7d2a705da177, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/e4312df7fdb04d74b6a119156a1e1685, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/926a1d5785144b4783b1224640ff7581, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/12fe1a6a4b79499ca7558b3e1cf5421a, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/00a3eb90e67d4fa9adb2e9841012c4e0, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/cc0a064112ef4de5bba522b015032887, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/1d72fab54415439fbe236e8e25012ee6, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/017f5663b65f432eb9c959f973e9cbe0, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/b1f06794470843268a4a21f19fb38ff5, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/01481e3b16bf439fbfc63e37b33733d0, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/e7c5daf5a9c44f4e910cba69a4c97d55, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/7aa789a8fabf4cc586e926b1d7e75856, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/f8d5e50acaeb4d8b9379202ce1469830, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/c193eb6a766e4e6fb90cdf47a6f9a81a, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/658303b5f67e46c98ae1ecc98ca89592, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/385a4d80b7b345a8a875d4d7e2c95e4a, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/885c6865791d4036a35b56549bb81901, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/93f897978079415082b8d01880747086, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/4a0d633a6ca14844925fd937da30fc1e, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/9a844e5165f14aca9ada2f473349aaf0, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/365ea5ede1e94dada6f1176d0c6190f3, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/77aaf68bce0a4a39b39aba633af90cf2, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/396aa1b0546f4fc8bd9b798c1ac99e7c, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/a9b92a2e33a444039ee2d96f32e73b7a, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/b746c6c37ca249979c59d4f35a07b458, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/b02c9efc2be04ae3a5762c35d56962fe, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/8f5a2dce2ec74cb880aedf8b2feef72b, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/ff563ed316b747ed9fe2f0d6fbc37028] to archive 2023-04-09 09:56:30,634 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-09 09:56:30,635 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/74b917e760f448caa7aad394f8a80885 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/74b917e760f448caa7aad394f8a80885 2023-04-09 09:56:30,635 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/c8b220063d05497ca6ca47ea08124297 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/c8b220063d05497ca6ca47ea08124297 2023-04-09 09:56:30,636 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/d88ac10f0b5a4045a38d2e5441f5c8f0 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/d88ac10f0b5a4045a38d2e5441f5c8f0 2023-04-09 09:56:30,637 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/9e49d1dbf7ca4ffc87d8cbfaa3cf445e to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/9e49d1dbf7ca4ffc87d8cbfaa3cf445e 2023-04-09 09:56:30,638 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/8e798f5fa9b5494a8e5e449294b29eae to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/8e798f5fa9b5494a8e5e449294b29eae 2023-04-09 09:56:30,638 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/9bde71d48cd34d158104f08f38f88121 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/9bde71d48cd34d158104f08f38f88121 2023-04-09 09:56:30,639 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/297f6d7fced140c9ac8a7d2a705da177 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/297f6d7fced140c9ac8a7d2a705da177 2023-04-09 09:56:30,640 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/e4312df7fdb04d74b6a119156a1e1685 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/e4312df7fdb04d74b6a119156a1e1685 2023-04-09 09:56:30,641 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/926a1d5785144b4783b1224640ff7581 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/926a1d5785144b4783b1224640ff7581 2023-04-09 09:56:30,642 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/12fe1a6a4b79499ca7558b3e1cf5421a to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/12fe1a6a4b79499ca7558b3e1cf5421a 2023-04-09 09:56:30,643 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/00a3eb90e67d4fa9adb2e9841012c4e0 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/00a3eb90e67d4fa9adb2e9841012c4e0 2023-04-09 09:56:30,645 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/cc0a064112ef4de5bba522b015032887 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/cc0a064112ef4de5bba522b015032887 2023-04-09 09:56:30,646 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/1d72fab54415439fbe236e8e25012ee6 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/1d72fab54415439fbe236e8e25012ee6 2023-04-09 09:56:30,647 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/017f5663b65f432eb9c959f973e9cbe0 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/017f5663b65f432eb9c959f973e9cbe0 2023-04-09 09:56:30,654 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/b1f06794470843268a4a21f19fb38ff5 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/b1f06794470843268a4a21f19fb38ff5 2023-04-09 09:56:30,655 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/01481e3b16bf439fbfc63e37b33733d0 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/01481e3b16bf439fbfc63e37b33733d0 2023-04-09 09:56:30,658 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/e7c5daf5a9c44f4e910cba69a4c97d55 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/e7c5daf5a9c44f4e910cba69a4c97d55 2023-04-09 09:56:30,660 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/7aa789a8fabf4cc586e926b1d7e75856 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/7aa789a8fabf4cc586e926b1d7e75856 2023-04-09 09:56:30,662 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/f8d5e50acaeb4d8b9379202ce1469830 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/f8d5e50acaeb4d8b9379202ce1469830 2023-04-09 09:56:30,663 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/c193eb6a766e4e6fb90cdf47a6f9a81a to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/c193eb6a766e4e6fb90cdf47a6f9a81a 2023-04-09 09:56:30,664 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/658303b5f67e46c98ae1ecc98ca89592 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/658303b5f67e46c98ae1ecc98ca89592 2023-04-09 09:56:30,665 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/385a4d80b7b345a8a875d4d7e2c95e4a to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/385a4d80b7b345a8a875d4d7e2c95e4a 2023-04-09 09:56:30,666 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/885c6865791d4036a35b56549bb81901 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/885c6865791d4036a35b56549bb81901 2023-04-09 09:56:30,667 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/93f897978079415082b8d01880747086 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/93f897978079415082b8d01880747086 2023-04-09 09:56:30,668 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/4a0d633a6ca14844925fd937da30fc1e to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/4a0d633a6ca14844925fd937da30fc1e 2023-04-09 09:56:30,669 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/9a844e5165f14aca9ada2f473349aaf0 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/9a844e5165f14aca9ada2f473349aaf0 2023-04-09 09:56:30,670 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/365ea5ede1e94dada6f1176d0c6190f3 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/365ea5ede1e94dada6f1176d0c6190f3 2023-04-09 09:56:30,671 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/77aaf68bce0a4a39b39aba633af90cf2 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/77aaf68bce0a4a39b39aba633af90cf2 2023-04-09 09:56:30,672 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/396aa1b0546f4fc8bd9b798c1ac99e7c to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/396aa1b0546f4fc8bd9b798c1ac99e7c 2023-04-09 09:56:30,673 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/a9b92a2e33a444039ee2d96f32e73b7a to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/a9b92a2e33a444039ee2d96f32e73b7a 2023-04-09 09:56:30,675 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/b746c6c37ca249979c59d4f35a07b458 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/b746c6c37ca249979c59d4f35a07b458 2023-04-09 09:56:30,682 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/b02c9efc2be04ae3a5762c35d56962fe to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/b02c9efc2be04ae3a5762c35d56962fe 2023-04-09 09:56:30,684 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/8f5a2dce2ec74cb880aedf8b2feef72b to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/8f5a2dce2ec74cb880aedf8b2feef72b 2023-04-09 09:56:30,685 DEBUG [StoreCloser-TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/ff563ed316b747ed9fe2f0d6fbc37028 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/ff563ed316b747ed9fe2f0d6fbc37028 2023-04-09 09:56:30,690 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/recovered.edits/792.seqid, newMaxSeqId=792, maxSeqId=4 2023-04-09 09:56:30,692 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95. 2023-04-09 09:56:30,692 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for c8127616cf5355dd7f1db99764192b95: 2023-04-09 09:56:30,693 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed c8127616cf5355dd7f1db99764192b95 2023-04-09 09:56:30,693 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=c8127616cf5355dd7f1db99764192b95, regionState=CLOSED 2023-04-09 09:56:30,694 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681034190693"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681034190693"}]},"ts":"1681034190693"} 2023-04-09 09:56:30,696 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-04-09 09:56:30,696 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; CloseRegionProcedure c8127616cf5355dd7f1db99764192b95, server=jenkins-hbase4.apache.org,42467,1681034019544 in 2.1840 sec 2023-04-09 09:56:30,697 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=63, resume processing ppid=62 2023-04-09 09:56:30,697 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=63, ppid=62, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=c8127616cf5355dd7f1db99764192b95, UNASSIGN in 2.1890 sec 2023-04-09 09:56:30,697 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681034190697"}]},"ts":"1681034190697"} 2023-04-09 09:56:30,698 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-09 09:56:30,699 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-09 09:56:30,701 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.1980 sec 2023-04-09 09:56:32,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-09 09:56:32,610 INFO [Listener at localhost/44563] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 62 completed 2023-04-09 09:56:32,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-09 09:56:32,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] procedure2.ProcedureExecutor(1029): Stored pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-09 09:56:32,612 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-09 09:56:32,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-09 09:56:32,613 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=65, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-09 09:56:32,618 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95 2023-04-09 09:56:32,620 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A, FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B, FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C, FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/recovered.edits] 2023-04-09 09:56:32,622 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/13c9425c5df4470c9d1e90b0c786ff5b to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/13c9425c5df4470c9d1e90b0c786ff5b 2023-04-09 09:56:32,623 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/249d084fc2bb4ab3b97ae45d6c6ca520 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/249d084fc2bb4ab3b97ae45d6c6ca520 2023-04-09 09:56:32,624 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/4decbb2e31ff4056b34a1463a88a5775 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/4decbb2e31ff4056b34a1463a88a5775 2023-04-09 09:56:32,625 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/74641726bb6748d7a697ff4a15ad44f4 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/74641726bb6748d7a697ff4a15ad44f4 2023-04-09 09:56:32,626 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/9be67c50a646452c88cfd8a7cd051680 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/9be67c50a646452c88cfd8a7cd051680 2023-04-09 09:56:32,626 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/dcd27f8800664591bdd5b262715d7379 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/A/dcd27f8800664591bdd5b262715d7379 2023-04-09 09:56:32,628 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/68153dba05a247df92e07774383d0c38 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/68153dba05a247df92e07774383d0c38 2023-04-09 09:56:32,629 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/70e66992b9734d4181587653dfc9ad6e to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/70e66992b9734d4181587653dfc9ad6e 2023-04-09 09:56:32,630 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/b24bb288dfc3453abc3b1f8b3ac15fe8 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/b24bb288dfc3453abc3b1f8b3ac15fe8 2023-04-09 09:56:32,631 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/b8c419a2881b46d4917ab7d87edc86cb to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/b8c419a2881b46d4917ab7d87edc86cb 2023-04-09 09:56:32,632 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/c1b6b91c2ea94a30834cd652ca578b82 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/c1b6b91c2ea94a30834cd652ca578b82 2023-04-09 09:56:32,632 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/fe7f706c60c8462884b34e4a7101d553 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/B/fe7f706c60c8462884b34e4a7101d553 2023-04-09 09:56:32,634 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/19e943f047004a5b8506e3e0f286c913 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/19e943f047004a5b8506e3e0f286c913 2023-04-09 09:56:32,635 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/4c855bad02974f2fac9c964e59566c9f to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/4c855bad02974f2fac9c964e59566c9f 2023-04-09 09:56:32,635 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/54e2d00e052d403db49d420f23c36873 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/54e2d00e052d403db49d420f23c36873 2023-04-09 09:56:32,636 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/6a7f2e15a74d4864830c4623d612379a to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/6a7f2e15a74d4864830c4623d612379a 2023-04-09 09:56:32,637 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/ac5a6cf4169f4d4595746db36350c98c to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/ac5a6cf4169f4d4595746db36350c98c 2023-04-09 09:56:32,638 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/b39e5ad44af349deb6d9a81773615b58 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/C/b39e5ad44af349deb6d9a81773615b58 2023-04-09 09:56:32,640 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/recovered.edits/792.seqid to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95/recovered.edits/792.seqid 2023-04-09 09:56:32,640 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/.tmp/data/default/TestAcidGuarantees/c8127616cf5355dd7f1db99764192b95 2023-04-09 09:56:32,640 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-09 09:56:32,640 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-09 09:56:32,641 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-09 09:56:32,643 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304090228e87b0c82426aa6ddf8afdbf5f218 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304090228e87b0c82426aa6ddf8afdbf5f218 2023-04-09 09:56:32,644 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040907ee4fd54a484662a5eaabd2d055c2f0 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040907ee4fd54a484662a5eaabd2d055c2f0 2023-04-09 09:56:32,644 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304090fce5e7cf9f2491f9346e94e9bfdd740 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304090fce5e7cf9f2491f9346e94e9bfdd740 2023-04-09 09:56:32,645 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304091660084efe574ada95722cf9e78850f6 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304091660084efe574ada95722cf9e78850f6 2023-04-09 09:56:32,646 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304091f9833f5a670455893c11217db309c75 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304091f9833f5a670455893c11217db309c75 2023-04-09 09:56:32,647 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040920b740b5f6bc47d58e7a4e7ba8860fb5 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040920b740b5f6bc47d58e7a4e7ba8860fb5 2023-04-09 09:56:32,647 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304092671355463a64ebe87903c47c83f8b4b to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304092671355463a64ebe87903c47c83f8b4b 2023-04-09 09:56:32,648 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304093f95f5fee39c46d7bb36141b60e59959 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304093f95f5fee39c46d7bb36141b60e59959 2023-04-09 09:56:32,649 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409401b6bf5f79a4ef687039407ed3efcbc to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409401b6bf5f79a4ef687039407ed3efcbc 2023-04-09 09:56:32,650 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409482c01af9b114e0eb62c1124b26a5e4c to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409482c01af9b114e0eb62c1124b26a5e4c 2023-04-09 09:56:32,651 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304094ad77b4095534703a2cd0cc4b91749a5 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304094ad77b4095534703a2cd0cc4b91749a5 2023-04-09 09:56:32,651 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304094ffc28f70a0542578b98b3146ae2602c to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304094ffc28f70a0542578b98b3146ae2602c 2023-04-09 09:56:32,652 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304095b00c3ee0c2e48a69ea4e420fa9b49b0 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304095b00c3ee0c2e48a69ea4e420fa9b49b0 2023-04-09 09:56:32,653 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040971157784980d4e34ae71acad650e9862 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040971157784980d4e34ae71acad650e9862 2023-04-09 09:56:32,654 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040974d4e1d97a1e4587bc463c62e9a69f32 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040974d4e1d97a1e4587bc463c62e9a69f32 2023-04-09 09:56:32,654 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040976d6cdc2873d4c2bba922b6d52d87d07 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040976d6cdc2873d4c2bba922b6d52d87d07 2023-04-09 09:56:32,655 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409812a8b1854d84c5eb889f66ea1603e32 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409812a8b1854d84c5eb889f66ea1603e32 2023-04-09 09:56:32,656 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304098ddbb6ff68cc46beaf026d94d4554d7f to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304098ddbb6ff68cc46beaf026d94d4554d7f 2023-04-09 09:56:32,657 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040991db2eb18c5f4ae3ac0b48527f5a0171 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040991db2eb18c5f4ae3ac0b48527f5a0171 2023-04-09 09:56:32,658 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409b030095902e545f69a6a20aaa41cbf5d to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409b030095902e545f69a6a20aaa41cbf5d 2023-04-09 09:56:32,659 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409b2d2a87e8b5b477d87ec8a834ffa94fe to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409b2d2a87e8b5b477d87ec8a834ffa94fe 2023-04-09 09:56:32,660 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409b3647897a3a740cba2903a6914b9e35e to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409b3647897a3a740cba2903a6914b9e35e 2023-04-09 09:56:32,660 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409ba0475327b90422b8adce979a114a25b to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409ba0475327b90422b8adce979a114a25b 2023-04-09 09:56:32,661 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409ba43f49e6cc34dab892436ae1591564c to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409ba43f49e6cc34dab892436ae1591564c 2023-04-09 09:56:32,662 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409bbee2b1833634b2e92ec5b6083b8b666 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409bbee2b1833634b2e92ec5b6083b8b666 2023-04-09 09:56:32,663 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409bcb8a376a560429190e6da5118abb952 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409bcb8a376a560429190e6da5118abb952 2023-04-09 09:56:32,663 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409cdf0bfadc03346b8a70e971265f354dc to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409cdf0bfadc03346b8a70e971265f354dc 2023-04-09 09:56:32,664 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409d9d45b4abe224a52a4c1667eee8003aa to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409d9d45b4abe224a52a4c1667eee8003aa 2023-04-09 09:56:32,665 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409dde041b0a55442b1b647c1463108bbc4 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409dde041b0a55442b1b647c1463108bbc4 2023-04-09 09:56:32,666 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409f0e73684a70f415bbc2ee3037e632477 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409f0e73684a70f415bbc2ee3037e632477 2023-04-09 09:56:32,667 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409fee514a95677427bb58f5eb66d4063b1 to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409fee514a95677427bb58f5eb66d4063b1 2023-04-09 09:56:32,668 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409fef7e78651dd4a70af95af6b9f4963bb to hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230409fef7e78651dd4a70af95af6b9f4963bb 2023-04-09 09:56:32,669 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-09 09:56:32,670 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=65, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-09 09:56:32,672 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-09 09:56:32,673 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-09 09:56:32,674 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=65, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-09 09:56:32,674 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-09 09:56:32,674 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681034192674"}]},"ts":"9223372036854775807"} 2023-04-09 09:56:32,676 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-09 09:56:32,676 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => c8127616cf5355dd7f1db99764192b95, NAME => 'TestAcidGuarantees,,1681034163959.c8127616cf5355dd7f1db99764192b95.', STARTKEY => '', ENDKEY => ''}] 2023-04-09 09:56:32,677 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-09 09:56:32,677 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681034192677"}]},"ts":"9223372036854775807"} 2023-04-09 09:56:32,678 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-09 09:56:32,679 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=65, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-09 09:56:32,680 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=65, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 69 msec 2023-04-09 09:56:32,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41863] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-09 09:56:32,714 INFO [Listener at localhost/44563] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 65 completed 2023-04-09 09:56:32,725 INFO [Listener at localhost/44563] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=228 (was 229), OpenFileDescriptor=592 (was 560) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=383 (was 374) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=1796 (was 1525) - AvailableMemoryMB LEAK? - 2023-04-09 09:56:32,726 INFO [Listener at localhost/44563] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-04-09 09:56:32,726 INFO [Listener at localhost/44563] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-09 09:56:32,726 DEBUG [Listener at localhost/44563] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6990bacd to 127.0.0.1:57816 2023-04-09 09:56:32,726 DEBUG [Listener at localhost/44563] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-09 09:56:32,727 DEBUG [Listener at localhost/44563] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-04-09 09:56:32,727 DEBUG [Listener at localhost/44563] util.JVMClusterUtil(257): Found active master hash=1596708042, stopped=false 2023-04-09 09:56:32,727 INFO [Listener at localhost/44563] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,41863,1681034018311 2023-04-09 09:56:32,729 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-09 09:56:32,729 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-09 09:56:32,729 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-09 09:56:32,729 INFO [Listener at localhost/44563] procedure2.ProcedureExecutor(629): Stopping 2023-04-09 09:56:32,730 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-09 09:56:32,730 DEBUG [Listener at localhost/44563] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4aaf52c6 to 127.0.0.1:57816 2023-04-09 09:56:32,730 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-09 09:56:32,730 DEBUG [Listener at localhost/44563] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-09 09:56:32,730 INFO [Listener at localhost/44563] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,42467,1681034019544' ***** 2023-04-09 09:56:32,730 INFO [Listener at localhost/44563] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-04-09 09:56:32,730 INFO [RS:0;jenkins-hbase4:42467] regionserver.HeapMemoryManager(220): Stopping 2023-04-09 09:56:32,731 INFO [RS:0;jenkins-hbase4:42467] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-04-09 09:56:32,731 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-04-09 09:56:32,731 INFO [RS:0;jenkins-hbase4:42467] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-04-09 09:56:32,731 INFO [RS:0;jenkins-hbase4:42467] regionserver.HRegionServer(3304): Received CLOSE for e67a71a4b88e824863c78f9fe9bfd221 2023-04-09 09:56:32,732 INFO [RS:0;jenkins-hbase4:42467] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:32,732 DEBUG [RS:0;jenkins-hbase4:42467] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x033291d8 to 127.0.0.1:57816 2023-04-09 09:56:32,732 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing e67a71a4b88e824863c78f9fe9bfd221, disabling compactions & flushes 2023-04-09 09:56:32,732 DEBUG [RS:0;jenkins-hbase4:42467] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-09 09:56:32,732 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1681034022203.e67a71a4b88e824863c78f9fe9bfd221. 2023-04-09 09:56:32,733 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1681034022203.e67a71a4b88e824863c78f9fe9bfd221. 2023-04-09 09:56:32,733 INFO [RS:0;jenkins-hbase4:42467] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-04-09 09:56:32,733 INFO [RS:0;jenkins-hbase4:42467] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-04-09 09:56:32,733 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1681034022203.e67a71a4b88e824863c78f9fe9bfd221. after waiting 0 ms 2023-04-09 09:56:32,733 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1681034022203.e67a71a4b88e824863c78f9fe9bfd221. 2023-04-09 09:56:32,733 INFO [RS:0;jenkins-hbase4:42467] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-04-09 09:56:32,733 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing e67a71a4b88e824863c78f9fe9bfd221 1/1 column families, dataSize=78 B heapSize=488 B 2023-04-09 09:56:32,733 INFO [RS:0;jenkins-hbase4:42467] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-04-09 09:56:32,733 INFO [RS:0;jenkins-hbase4:42467] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-04-09 09:56:32,734 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-09 09:56:32,734 DEBUG [RS:0;jenkins-hbase4:42467] regionserver.HRegionServer(1479): Online Regions={e67a71a4b88e824863c78f9fe9bfd221=hbase:namespace,,1681034022203.e67a71a4b88e824863c78f9fe9bfd221., 1588230740=hbase:meta,,1.1588230740} 2023-04-09 09:56:32,734 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-09 09:56:32,734 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-09 09:56:32,734 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-09 09:56:32,734 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-09 09:56:32,734 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=20.92 KB heapSize=36.24 KB 2023-04-09 09:56:32,735 DEBUG [RS:0;jenkins-hbase4:42467] regionserver.HRegionServer(1505): Waiting on 1588230740, e67a71a4b88e824863c78f9fe9bfd221 2023-04-09 09:56:32,744 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/hbase/namespace/e67a71a4b88e824863c78f9fe9bfd221/.tmp/info/6b401b58dd644afa8ce5257db5e4768f 2023-04-09 09:56:32,748 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/hbase/namespace/e67a71a4b88e824863c78f9fe9bfd221/.tmp/info/6b401b58dd644afa8ce5257db5e4768f as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/hbase/namespace/e67a71a4b88e824863c78f9fe9bfd221/info/6b401b58dd644afa8ce5257db5e4768f 2023-04-09 09:56:32,752 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/hbase/namespace/e67a71a4b88e824863c78f9fe9bfd221/info/6b401b58dd644afa8ce5257db5e4768f, entries=2, sequenceid=6, filesize=4.8 K 2023-04-09 09:56:32,752 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for e67a71a4b88e824863c78f9fe9bfd221 in 19ms, sequenceid=6, compaction requested=false 2023-04-09 09:56:32,757 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/hbase/namespace/e67a71a4b88e824863c78f9fe9bfd221/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-04-09 09:56:32,758 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1681034022203.e67a71a4b88e824863c78f9fe9bfd221. 2023-04-09 09:56:32,758 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for e67a71a4b88e824863c78f9fe9bfd221: 2023-04-09 09:56:32,758 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1681034022203.e67a71a4b88e824863c78f9fe9bfd221. 2023-04-09 09:56:32,817 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-04-09 09:56:32,935 DEBUG [RS:0;jenkins-hbase4:42467] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-09 09:56:33,136 DEBUG [RS:0;jenkins-hbase4:42467] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-09 09:56:33,140 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=18.26 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/hbase/meta/1588230740/.tmp/info/a45026b5a41e496188ea4f7ca6d04860 2023-04-09 09:56:33,144 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for a45026b5a41e496188ea4f7ca6d04860 2023-04-09 09:56:33,284 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-04-09 09:56:33,284 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-04-09 09:56:33,336 DEBUG [RS:0;jenkins-hbase4:42467] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-09 09:56:33,536 DEBUG [RS:0;jenkins-hbase4:42467] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-09 09:56:33,549 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=588 B at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/hbase/meta/1588230740/.tmp/rep_barrier/8767601d711d44cdb7fcc3a2b860ec6f 2023-04-09 09:56:33,552 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 8767601d711d44cdb7fcc3a2b860ec6f 2023-04-09 09:56:33,737 INFO [RS:0;jenkins-hbase4:42467] regionserver.HRegionServer(1475): Waiting on 1 regions to close 2023-04-09 09:56:33,737 DEBUG [RS:0;jenkins-hbase4:42467] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740} 2023-04-09 09:56:33,737 DEBUG [RS:0;jenkins-hbase4:42467] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-09 09:56:33,937 DEBUG [RS:0;jenkins-hbase4:42467] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-09 09:56:33,958 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.08 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/hbase/meta/1588230740/.tmp/table/efb34f71c50745c097a2de0a66a46266 2023-04-09 09:56:33,961 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for efb34f71c50745c097a2de0a66a46266 2023-04-09 09:56:33,962 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/hbase/meta/1588230740/.tmp/info/a45026b5a41e496188ea4f7ca6d04860 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/hbase/meta/1588230740/info/a45026b5a41e496188ea4f7ca6d04860 2023-04-09 09:56:33,965 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for a45026b5a41e496188ea4f7ca6d04860 2023-04-09 09:56:33,965 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/hbase/meta/1588230740/info/a45026b5a41e496188ea4f7ca6d04860, entries=22, sequenceid=93, filesize=7.1 K 2023-04-09 09:56:33,966 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/hbase/meta/1588230740/.tmp/rep_barrier/8767601d711d44cdb7fcc3a2b860ec6f as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/hbase/meta/1588230740/rep_barrier/8767601d711d44cdb7fcc3a2b860ec6f 2023-04-09 09:56:33,969 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 8767601d711d44cdb7fcc3a2b860ec6f 2023-04-09 09:56:33,970 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/hbase/meta/1588230740/rep_barrier/8767601d711d44cdb7fcc3a2b860ec6f, entries=6, sequenceid=93, filesize=5.5 K 2023-04-09 09:56:33,970 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/hbase/meta/1588230740/.tmp/table/efb34f71c50745c097a2de0a66a46266 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/hbase/meta/1588230740/table/efb34f71c50745c097a2de0a66a46266 2023-04-09 09:56:33,973 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for efb34f71c50745c097a2de0a66a46266 2023-04-09 09:56:33,973 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/hbase/meta/1588230740/table/efb34f71c50745c097a2de0a66a46266, entries=9, sequenceid=93, filesize=5.5 K 2023-04-09 09:56:33,974 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.92 KB/21417, heapSize ~36.20 KB/37064, currentSize=0 B/0 for 1588230740 in 1240ms, sequenceid=93, compaction requested=false 2023-04-09 09:56:33,982 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/data/hbase/meta/1588230740/recovered.edits/96.seqid, newMaxSeqId=96, maxSeqId=1 2023-04-09 09:56:33,982 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-09 09:56:33,983 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-09 09:56:33,983 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-09 09:56:33,983 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-04-09 09:56:34,137 INFO [RS:0;jenkins-hbase4:42467] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,42467,1681034019544; all regions closed. 2023-04-09 09:56:34,145 DEBUG [RS:0;jenkins-hbase4:42467] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/oldWALs 2023-04-09 09:56:34,145 INFO [RS:0;jenkins-hbase4:42467] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C42467%2C1681034019544.meta:.meta(num 1681034021954) 2023-04-09 09:56:34,149 DEBUG [RS:0;jenkins-hbase4:42467] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/oldWALs 2023-04-09 09:56:34,149 INFO [RS:0;jenkins-hbase4:42467] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C42467%2C1681034019544:(num 1681034021396) 2023-04-09 09:56:34,149 DEBUG [RS:0;jenkins-hbase4:42467] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-09 09:56:34,149 INFO [RS:0;jenkins-hbase4:42467] regionserver.LeaseManager(133): Closed leases 2023-04-09 09:56:34,150 INFO [RS:0;jenkins-hbase4:42467] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-04-09 09:56:34,150 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-09 09:56:34,151 INFO [RS:0;jenkins-hbase4:42467] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:42467 2023-04-09 09:56:34,156 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,42467,1681034019544 2023-04-09 09:56:34,156 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-09 09:56:34,156 ERROR [Listener at localhost/44563-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@70a76ee9 rejected from java.util.concurrent.ThreadPoolExecutor@7498f425[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2408] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-09 09:56:34,156 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-09 09:56:34,157 ERROR [Listener at localhost/44563-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4b7adf83 rejected from java.util.concurrent.ThreadPoolExecutor@7498f425[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2408] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-09 09:56:34,157 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,42467,1681034019544] 2023-04-09 09:56:34,158 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,42467,1681034019544; numProcessing=1 2023-04-09 09:56:34,160 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,42467,1681034019544 already deleted, retry=false 2023-04-09 09:56:34,160 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,42467,1681034019544 expired; onlineServers=0 2023-04-09 09:56:34,160 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,41863,1681034018311' ***** 2023-04-09 09:56:34,160 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-04-09 09:56:34,160 DEBUG [M:0;jenkins-hbase4:41863] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1e16def1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-09 09:56:34,160 INFO [M:0;jenkins-hbase4:41863] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,41863,1681034018311 2023-04-09 09:56:34,160 INFO [M:0;jenkins-hbase4:41863] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,41863,1681034018311; all regions closed. 2023-04-09 09:56:34,160 DEBUG [M:0;jenkins-hbase4:41863] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-09 09:56:34,160 DEBUG [M:0;jenkins-hbase4:41863] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-04-09 09:56:34,160 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-04-09 09:56:34,160 DEBUG [M:0;jenkins-hbase4:41863] cleaner.HFileCleaner(317): Stopping file delete threads 2023-04-09 09:56:34,160 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1681034021096] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1681034021096,5,FailOnTimeoutGroup] 2023-04-09 09:56:34,160 INFO [M:0;jenkins-hbase4:41863] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-04-09 09:56:34,160 INFO [M:0;jenkins-hbase4:41863] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-04-09 09:56:34,160 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1681034021097] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1681034021097,5,FailOnTimeoutGroup] 2023-04-09 09:56:34,161 INFO [M:0;jenkins-hbase4:41863] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-04-09 09:56:34,161 DEBUG [M:0;jenkins-hbase4:41863] master.HMaster(1512): Stopping service threads 2023-04-09 09:56:34,161 INFO [M:0;jenkins-hbase4:41863] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-04-09 09:56:34,161 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-04-09 09:56:34,161 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-09 09:56:34,161 ERROR [M:0;jenkins-hbase4:41863] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[IPC Client (695307169) connection to localhost/127.0.0.1:46631 from jenkins,5,PEWorkerGroup] Thread[HFileArchiver-13,5,PEWorkerGroup] 2023-04-09 09:56:34,162 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-09 09:56:34,162 INFO [M:0;jenkins-hbase4:41863] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-09 09:56:34,162 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-04-09 09:56:34,162 DEBUG [M:0;jenkins-hbase4:41863] zookeeper.ZKUtil(398): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-04-09 09:56:34,162 WARN [M:0;jenkins-hbase4:41863] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-04-09 09:56:34,162 INFO [M:0;jenkins-hbase4:41863] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-09 09:56:34,163 INFO [M:0;jenkins-hbase4:41863] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-04-09 09:56:34,163 DEBUG [M:0;jenkins-hbase4:41863] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-09 09:56:34,163 INFO [M:0;jenkins-hbase4:41863] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-09 09:56:34,163 DEBUG [M:0;jenkins-hbase4:41863] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-09 09:56:34,163 DEBUG [M:0;jenkins-hbase4:41863] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-09 09:56:34,163 DEBUG [M:0;jenkins-hbase4:41863] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-09 09:56:34,163 INFO [M:0;jenkins-hbase4:41863] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=345.05 KB heapSize=393.75 KB 2023-04-09 09:56:34,172 INFO [M:0;jenkins-hbase4:41863] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=345.05 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/c9e12c916eae4ec1bc2690880d8434e2 2023-04-09 09:56:34,176 INFO [M:0;jenkins-hbase4:41863] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for c9e12c916eae4ec1bc2690880d8434e2 2023-04-09 09:56:34,177 DEBUG [M:0;jenkins-hbase4:41863] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/c9e12c916eae4ec1bc2690880d8434e2 as hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/c9e12c916eae4ec1bc2690880d8434e2 2023-04-09 09:56:34,179 INFO [M:0;jenkins-hbase4:41863] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for c9e12c916eae4ec1bc2690880d8434e2 2023-04-09 09:56:34,179 INFO [M:0;jenkins-hbase4:41863] regionserver.HStore(1080): Added hdfs://localhost:46631/user/jenkins/test-data/6fde6337-ebc6-e3d0-515d-8fa7f07a68dc/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/c9e12c916eae4ec1bc2690880d8434e2, entries=65, sequenceid=578, filesize=24.7 K 2023-04-09 09:56:34,180 INFO [M:0;jenkins-hbase4:41863] regionserver.HRegion(2948): Finished flush of dataSize ~345.05 KB/353331, heapSize ~393.73 KB/403184, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 17ms, sequenceid=578, compaction requested=false 2023-04-09 09:56:34,182 INFO [M:0;jenkins-hbase4:41863] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-09 09:56:34,182 DEBUG [M:0;jenkins-hbase4:41863] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-09 09:56:34,185 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-09 09:56:34,185 INFO [M:0;jenkins-hbase4:41863] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-04-09 09:56:34,185 INFO [M:0;jenkins-hbase4:41863] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:41863 2023-04-09 09:56:34,187 DEBUG [M:0;jenkins-hbase4:41863] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,41863,1681034018311 already deleted, retry=false 2023-04-09 09:56:34,258 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-09 09:56:34,258 INFO [RS:0;jenkins-hbase4:42467] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,42467,1681034019544; zookeeper connection closed. 2023-04-09 09:56:34,259 ERROR [Listener at localhost/44563-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@336bd5d6 rejected from java.util.concurrent.ThreadPoolExecutor@7498f425[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2408] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-09 09:56:34,259 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): regionserver:42467-0x101685764600001, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-09 09:56:34,259 ERROR [Listener at localhost/44563-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4c2ed0ef rejected from java.util.concurrent.ThreadPoolExecutor@7498f425[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2408] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-09 09:56:34,263 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@25adace0] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@25adace0 2023-04-09 09:56:34,263 INFO [Listener at localhost/44563] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2023-04-09 09:56:34,358 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-09 09:56:34,358 INFO [M:0;jenkins-hbase4:41863] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,41863,1681034018311; zookeeper connection closed. 2023-04-09 09:56:34,359 ERROR [Listener at localhost/44563-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@46087f78 rejected from java.util.concurrent.ThreadPoolExecutor@4aa25391[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4027] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-09 09:56:34,359 DEBUG [Listener at localhost/44563-EventThread] zookeeper.ZKWatcher(600): master:41863-0x101685764600000, quorum=127.0.0.1:57816, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-09 09:56:34,359 ERROR [Listener at localhost/44563-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@cb640a9 rejected from java.util.concurrent.ThreadPoolExecutor@4aa25391[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4027] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-09 09:56:34,360 WARN [Listener at localhost/44563] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-04-09 09:56:34,364 INFO [Listener at localhost/44563] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-09 09:56:34,473 WARN [BP-1087743619-172.31.14.131-1681034014990 heartbeating to localhost/127.0.0.1:46631] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-04-09 09:56:34,473 WARN [BP-1087743619-172.31.14.131-1681034014990 heartbeating to localhost/127.0.0.1:46631] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1087743619-172.31.14.131-1681034014990 (Datanode Uuid a694fe9c-ce8f-43d0-bbb2-d1fd0f9fcdee) service to localhost/127.0.0.1:46631 2023-04-09 09:56:34,476 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9114bfcc-42bd-9802-05ab-0c25bc62528e/cluster_581cb602-ccd9-7b40-d298-a4df8bdcd7d5/dfs/data/data1/current/BP-1087743619-172.31.14.131-1681034014990] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-09 09:56:34,477 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9114bfcc-42bd-9802-05ab-0c25bc62528e/cluster_581cb602-ccd9-7b40-d298-a4df8bdcd7d5/dfs/data/data2/current/BP-1087743619-172.31.14.131-1681034014990] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-09 09:56:34,518 INFO [Listener at localhost/44563] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-09 09:56:34,697 INFO [Listener at localhost/44563] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-04-09 09:56:34,893 INFO [Listener at localhost/44563] hbase.HBaseTestingUtility(1293): Minicluster is down